Interactive Statue Conversation App
Budget: $3,000 – $5,000 USD
I need a proof-of-concept mobile web application that works smoothly on both Android and iOS. The core idea is simple but technically rich: a visitor points their phone at a public monument, the camera captures an image, the app recognises the figures depicted, and then each of those historical characters speaks back in a distinct, period-appropriate voice.
Here is the flow I have in mind:
1. Vision: Use a lightweight on-device or edge computer-vision model to identify every figure in the photo. Accuracy, speed, and the ability to handle outdoor lighting are crucial. No text recognition is required—just figure identification.
2. Voice design: For each identified figure, generate an era-authentic voice with ElevenLabs’ Voice Design API. Think marble statues suddenly speaking in first-person tones that match their century.
3. Multi-voice orchestration: Feed the voices into an ElevenLabs Agent so multiple characters can converse naturally with the user and with one another. The agent needs to manage context, respond in real time, and switch seamlessly between speakers.
4. Real-time dialogue: Stream audio over WebRTC so latency stays low enough that the exchange feels like a live conversation. When the user speaks, transcribe locally (or with Whisper) and pass the text back to the agent for a response.
5. Optional research depth: If you can layer in quick web look-ups to enrich the agent’s knowledge about lesser-known statues, that would be a plus, but keep the main loop fast.
Acceptance criteria
• A single mobile-friendly web URL that I can open on both Android and iOS browsers.
• Take/choose a photo of a statue, identify figures, and show their names on screen.
• Tap “Talk” and hold a natural voice conversation where each figure replies in its own, era-authentic voice with round-trip latency under two seconds.
• Source code, environment setup notes, and a short README that explains how to swap out models or keys.
If you have prior experience with ElevenLabs, WebRTC, or on-device vision, let me know—that will speed up the build. We have the directions how to make the app
Here is the flow I have in mind:
1. Vision: Use a lightweight on-device or edge computer-vision model to identify every figure in the photo. Accuracy, speed, and the ability to handle outdoor lighting are crucial. No text recognition is required—just figure identification.
2. Voice design: For each identified figure, generate an era-authentic voice with ElevenLabs’ Voice Design API. Think marble statues suddenly speaking in first-person tones that match their century.
3. Multi-voice orchestration: Feed the voices into an ElevenLabs Agent so multiple characters can converse naturally with the user and with one another. The agent needs to manage context, respond in real time, and switch seamlessly between speakers.
4. Real-time dialogue: Stream audio over WebRTC so latency stays low enough that the exchange feels like a live conversation. When the user speaks, transcribe locally (or with Whisper) and pass the text back to the agent for a response.
5. Optional research depth: If you can layer in quick web look-ups to enrich the agent’s knowledge about lesser-known statues, that would be a plus, but keep the main loop fast.
Acceptance criteria
• A single mobile-friendly web URL that I can open on both Android and iOS browsers.
• Take/choose a photo of a statue, identify figures, and show their names on screen.
• Tap “Talk” and hold a natural voice conversation where each figure replies in its own, era-authentic voice with round-trip latency under two seconds.
• Source code, environment setup notes, and a short README that explains how to swap out models or keys.
If you have prior experience with ElevenLabs, WebRTC, or on-device vision, let me know—that will speed up the build. We have the directions how to make the app
Related categories:
Mobile App Development
iPhone
Android
Objective C
Computer Vision
WebRTC
ElevenLabs
Image Recognition