AI Agent Engineer - Python Backend Engineer to Diagnose OpenAI Realtime + Twilio Media Streams Issues (FastAPI, WebSockets)
Budget: ₹12,500 – ₹37,500 INR
Overview
I’m building Vozlia, a voice assistant platform. The system handles phone calls via Twilio, streams audio via Twilio Media Streams (WebSocket) to a FastAPI backend, and connects to OpenAI Realtime API for live speech interaction and tool routing (Gmail skill via FSM/router). I need a strong Python engineer to diagnose and stabilize realtime call flows and eliminate recurring production issues (deploy failures, WS lifecycle bugs, audio buffer/commit timing problems, response state issues).
This is a debugging + stabilization contract, not a greenfield build.
Current Architecture (High Level)
Twilio Voice inbound webhook returns TwiML <Connect><Stream> to backend WebSocket endpoint
Backend is FastAPI running on Render
Backend WebSocket receives Twilio audio frames, forwards to OpenAI Realtime WebSocket
Backend streams assistant audio back to Twilio
FSM/router handles “skills” requests (Gmail summaries, etc.), while general chat remains realtime
Primary Goals (What you will deliver)
Root-cause analysis of current failures and inconsistent behavior
Stabilize WebSocket lifecycle (Twilio ↔ backend ↔ OpenAI Realtime)
Fix or redesign buffering/commit logic so audio is consistently intelligible and responses always trigger
Ensure FSM-driven “skill speech” reliably plays on the call (no silent call after chitchat → email request)
Add battle-tested logging + correlation IDs so failures are diagnosable in minutes, not hours
Produce a short runbook: how to reproduce, debug, deploy safely
Known Symptoms / Issues to Investigate
Deploy failures from missing imports / undefined settings / refactor regressions
Twilio call connects but later goes silent after certain intent transitions (e.g., chitchat then “read my email”)
OpenAI Realtime errors like committing audio too frequently / too small (e.g., “commit empty” style failures)
Response lifecycle bugs: active_response_id not cleared → backend skips new response.create
“FSM speech enqueued” vs “speech actually sent to Twilio” mismatch
Occasional non-fatal cancel errors (cancel_not_active), but state may drift
Need consistent routing between general realtime replies and skill-driven replies
Tech Stack / Libraries (must be comfortable with most of these)
Python
fastapi
uvicorn
websockets (or websocket-client depending on implementation)
httpx
pydantic
asyncio
Telephony
twilio (TwiML + Media Streams)
OpenAI
OpenAI Realtime API via WebSocket
openai python SDK (if used elsewhere)
DB / Config (likely present)
sqlalchemy
environment variables / pydantic-settings (or equivalent)
Ops / Debug
structured logging (Python logging / JSON logs)
(bonus) Sentry, OpenTelemetry, Prometheus metrics
What You Must Do (Task Breakdown)
Phase 1 — Audit & Reproduction
Pull repo and deploy locally (or in a staging service)
Reproduce at least 2 failure modes from call logs
Identify root cause(s) with evidence: logs, timing, WS state machine, commit intervals, etc.
Phase 2 — Fixes & Stabilization
Refactor WebSocket code into a clear state machine:
connected / streaming / responding / interrupted / closed
Implement safe audio buffering strategy:
commit only on valid frame thresholds (avoid <100ms commits)
guardrails for empty commits
Ensure response lifecycle is correct:
always clear active_response_id on completion/cancel/error paths
prevent “stuck response” blocking future responses
Ensure skill responses are delivered reliably:
“FSM speech sent to Twilio” must be verifiably executed
no silent failures when switching modes mid-call
Phase 3 — Observability & Runbook
Add correlation IDs per call (CallSid-based) across all logs
Add explicit logs for:
WS events from Twilio
WS events to/from OpenAI
audio frames in/out stats
response.create/cancel/complete transitions
FSM routing triggers and final speech output to Twilio
Produce a runbook:
reproduction steps
how to read logs
common failure patterns and fixes
deploy checklist
Deliverables
PR(s) with clean, documented fixes
A short “Stabilization Report” describing:
what was broken
how it was fixed
what to watch for
A Runbook for ongoing maintenance
Qualification Requirements (Please don’t apply if you don’t match)
Strong asyncio + WebSocket debugging experience in production
Proven experience with FastAPI backend systems
Experience with Twilio Voice + Media Streams (or similar streaming telephony) strongly preferred
Comfortable debugging real-time audio streaming / buffering / latency issues
Able to read logs and trace timing/state bugs without guesswork
Screening Questions (Answer these in your proposal)
Describe a real project where you debugged a WebSocket/asyncio system in production. What was the root cause and fix?
Have you worked with Twilio Media Streams? If yes, what issues did you face (timing, audio encoding, WS reconnect, etc.)?
How would you prevent “stuck state” in a system tracking an active response ID?
What’s your approach to diagnosing “audio goes silent after mode switch”?
What tools do you use for observability (structured logs, tracing, etc.)?
Access / Info I Will Provide
GitHub repo access
Render logs
Twilio call SIDs + reproduction script (“call in, say X, then say Y”)
Environment variable list (secrets redacted; you’ll get them securely)
Engagement Type
Short contract focused on stabilization (can extend if successful)
I’m building Vozlia, a voice assistant platform. The system handles phone calls via Twilio, streams audio via Twilio Media Streams (WebSocket) to a FastAPI backend, and connects to OpenAI Realtime API for live speech interaction and tool routing (Gmail skill via FSM/router). I need a strong Python engineer to diagnose and stabilize realtime call flows and eliminate recurring production issues (deploy failures, WS lifecycle bugs, audio buffer/commit timing problems, response state issues).
This is a debugging + stabilization contract, not a greenfield build.
Current Architecture (High Level)
Twilio Voice inbound webhook returns TwiML <Connect><Stream> to backend WebSocket endpoint
Backend is FastAPI running on Render
Backend WebSocket receives Twilio audio frames, forwards to OpenAI Realtime WebSocket
Backend streams assistant audio back to Twilio
FSM/router handles “skills” requests (Gmail summaries, etc.), while general chat remains realtime
Primary Goals (What you will deliver)
Root-cause analysis of current failures and inconsistent behavior
Stabilize WebSocket lifecycle (Twilio ↔ backend ↔ OpenAI Realtime)
Fix or redesign buffering/commit logic so audio is consistently intelligible and responses always trigger
Ensure FSM-driven “skill speech” reliably plays on the call (no silent call after chitchat → email request)
Add battle-tested logging + correlation IDs so failures are diagnosable in minutes, not hours
Produce a short runbook: how to reproduce, debug, deploy safely
Known Symptoms / Issues to Investigate
Deploy failures from missing imports / undefined settings / refactor regressions
Twilio call connects but later goes silent after certain intent transitions (e.g., chitchat then “read my email”)
OpenAI Realtime errors like committing audio too frequently / too small (e.g., “commit empty” style failures)
Response lifecycle bugs: active_response_id not cleared → backend skips new response.create
“FSM speech enqueued” vs “speech actually sent to Twilio” mismatch
Occasional non-fatal cancel errors (cancel_not_active), but state may drift
Need consistent routing between general realtime replies and skill-driven replies
Tech Stack / Libraries (must be comfortable with most of these)
Python
fastapi
uvicorn
websockets (or websocket-client depending on implementation)
httpx
pydantic
asyncio
Telephony
twilio (TwiML + Media Streams)
OpenAI
OpenAI Realtime API via WebSocket
openai python SDK (if used elsewhere)
DB / Config (likely present)
sqlalchemy
environment variables / pydantic-settings (or equivalent)
Ops / Debug
structured logging (Python logging / JSON logs)
(bonus) Sentry, OpenTelemetry, Prometheus metrics
What You Must Do (Task Breakdown)
Phase 1 — Audit & Reproduction
Pull repo and deploy locally (or in a staging service)
Reproduce at least 2 failure modes from call logs
Identify root cause(s) with evidence: logs, timing, WS state machine, commit intervals, etc.
Phase 2 — Fixes & Stabilization
Refactor WebSocket code into a clear state machine:
connected / streaming / responding / interrupted / closed
Implement safe audio buffering strategy:
commit only on valid frame thresholds (avoid <100ms commits)
guardrails for empty commits
Ensure response lifecycle is correct:
always clear active_response_id on completion/cancel/error paths
prevent “stuck response” blocking future responses
Ensure skill responses are delivered reliably:
“FSM speech sent to Twilio” must be verifiably executed
no silent failures when switching modes mid-call
Phase 3 — Observability & Runbook
Add correlation IDs per call (CallSid-based) across all logs
Add explicit logs for:
WS events from Twilio
WS events to/from OpenAI
audio frames in/out stats
response.create/cancel/complete transitions
FSM routing triggers and final speech output to Twilio
Produce a runbook:
reproduction steps
how to read logs
common failure patterns and fixes
deploy checklist
Deliverables
PR(s) with clean, documented fixes
A short “Stabilization Report” describing:
what was broken
how it was fixed
what to watch for
A Runbook for ongoing maintenance
Qualification Requirements (Please don’t apply if you don’t match)
Strong asyncio + WebSocket debugging experience in production
Proven experience with FastAPI backend systems
Experience with Twilio Voice + Media Streams (or similar streaming telephony) strongly preferred
Comfortable debugging real-time audio streaming / buffering / latency issues
Able to read logs and trace timing/state bugs without guesswork
Screening Questions (Answer these in your proposal)
Describe a real project where you debugged a WebSocket/asyncio system in production. What was the root cause and fix?
Have you worked with Twilio Media Streams? If yes, what issues did you face (timing, audio encoding, WS reconnect, etc.)?
How would you prevent “stuck state” in a system tracking an active response ID?
What’s your approach to diagnosing “audio goes silent after mode switch”?
What tools do you use for observability (structured logs, tracing, etc.)?
Access / Info I Will Provide
GitHub repo access
Render logs
Twilio call SIDs + reproduction script (“call in, say X, then say Y”)
Environment variable list (secrets redacted; you’ll get them securely)
Engagement Type
Short contract focused on stabilization (can extend if successful)