Python Real-Time Voice Agent
Budget: $250 – $750 USD
I need a fully-async Python 3.11+ pipeline that turns a live phone call into a smooth, sub-2.5 s p50 conversational loop.
Here’s the flow I have in mind: a Twilio SIP trunk delivers the RTP stream to a WebSocket bridge; LiveKit Agents SDK manages the media session; Deepgram Nova-3 handles streaming STT; the running transcript feeds Claude (with tool-use enabled); Claude’s text comes back out through ElevenLabs Flash v2.5 for TTS and streams to the caller in real time.
What I need from you is a working reference implementation, instrumented and tuned so I can see latency at each hop and fine-tune Voice Activity Detection thresholds. The code must retry transient errors, log everything that matters, notify the caller gracefully on trouble, and, if the Sonnet tier fails mid-call, fall back to the Haiku model without dropping the line.
Deliverables
• Docker-ready Python project with clear README
• End-to-end demo showing p50 latency < 2.5 s on a five-minute call
• VAD parameters exposed via config file or CLI flag
• Prometheus/Grafana-friendly metrics for STT, LLM, TTS, and network hops
• Structured error-handling module covering retry, notify, log, and Haiku fallback paths
• Brief write-up of tuning choices and next-step recommendations
If this sounds like your kind of build, let’s get the call loop humming.
Here’s the flow I have in mind: a Twilio SIP trunk delivers the RTP stream to a WebSocket bridge; LiveKit Agents SDK manages the media session; Deepgram Nova-3 handles streaming STT; the running transcript feeds Claude (with tool-use enabled); Claude’s text comes back out through ElevenLabs Flash v2.5 for TTS and streams to the caller in real time.
What I need from you is a working reference implementation, instrumented and tuned so I can see latency at each hop and fine-tune Voice Activity Detection thresholds. The code must retry transient errors, log everything that matters, notify the caller gracefully on trouble, and, if the Sonnet tier fails mid-call, fall back to the Haiku model without dropping the line.
Deliverables
• Docker-ready Python project with clear README
• End-to-end demo showing p50 latency < 2.5 s on a five-minute call
• VAD parameters exposed via config file or CLI flag
• Prometheus/Grafana-friendly metrics for STT, LLM, TTS, and network hops
• Structured error-handling module covering retry, notify, log, and Haiku fallback paths
• Brief write-up of tuning choices and next-step recommendations
If this sounds like your kind of build, let’s get the call loop humming.