Real-Time Voice AI System Optimization
Budget: ₹1,500 – ₹12,500 INR
Here’s the revised version focused entirely on fixing and optimizing your existing custom architecture without migrating to LiveKit.
# Senior Real-Time Voice AI Engineer Needed (SIP + Streaming STT/TTS + Low Latency + Barge-In)
## Project Overview
We are building a real-time AI voice calling platform for automated outbound/inbound calls that can schedule meetings and handle natural human-like conversations over phone calls.
Current stack:
* Frontend: React
* Backend: Express.js / Node.js
* SIP Provider: Vobiz SIP
* STT/TTS: Sarvam AI
* LLM: Currently Groq API (may migrate later)
* Existing architecture: Custom-built real-time voice pipeline
The system is already functional and capable of making calls, but we need an experienced real-time voice AI engineer to optimize the architecture and solve latency + interruption handling problems.
---
# Current Issues
## 1. High Response Latency (4–7 seconds)
Current flow:
STT → LLM → TTS
Problems:
* AI takes too long to respond
* System waits for full LLM response before TTS
* Audio generation is not properly streamed
* Calls feel unnatural due to response delay
---
## 2. Barge-In / Interruption Problems
We partially implemented interruption handling, but conversation flow still breaks.
Current issues:
* User interrupts AI while it is speaking
* Existing pipeline becomes locked/busy
* Old utterances replay later
* Conversation becomes out-of-sync
* Replay-loop issue:
"Pipeline busy → buffering utterance for replay"
We need proper:
* Cancel-and-restart architecture
* Real-time interruption handling
* Audio buffer flushing
* Immediate TTS stop on user speech
---
## 3. Real-Time Streaming Pipeline Optimization
Need improvements in:
* Streaming STT
* Streaming LLM responses
* Streaming TTS audio
* Turn detection
* Voice Activity Detection (VAD)
* Concurrent session handling
* SIP audio streaming
* Memory cleanup
* Queue management
* Event listener cleanup
---
# Technical Requirements
We are specifically looking for someone with experience in:
* Real-time conversational AI
* Voice calling systems
* SIP/WebRTC architectures
* Streaming audio pipelines
* Low-latency AI systems
* Real-time Node.js systems
* Telephony integrations
* AI interruption/barge-in systems
Strong experience with:
* SIP / RTP
* WebSocket streaming
* Node.js streams
* Real-time media handling
* STT/TTS streaming APIs
* AI voice agents
* Express.js backend optimization
---
# Important Note
We are NOT looking to migrate to LiveKit or rebuild the platform using another framework.
We want to optimize and stabilize our EXISTING custom architecture and make it production-ready.
Please apply only if you are comfortable debugging and improving a custom real-time voice pipeline.
---
# Expected Deliverables
* Reduce response latency to near real-time
* Implement proper interruption/barge-in handling
* Remove replay-loop architecture issues
* Optimize streaming pipeline
* Improve TTS responsiveness
* Improve STT responsiveness
* Proper pipeline cancellation handling
* Stable concurrent call handling
* Production-ready architecture improvements
* Clean/refactored backend implementation
* Technical documentation
---
# Current Technical Problems Observed
Examples from logs:
* "Pipeline busy → buffering utterance for replay"
* Delayed TTS generation
* TTS cancellation issues
* Queued audio replay
* Long LLM wait times
* EventEmitter memory leak warnings
* In-flight request cancellation problems
---
# Ideal Candidate
You have previously worked on:
* AI calling systems
* Voice bots
* SIP-based applications
* Telephony AI
* Real-time audio streaming
* AI assistants with interruption support
* Low-latency media systems
---
# When Applying Please Include
1. Similar voice AI projects you worked on
2. Your experience with SIP/telephony systems
3. Experience with streaming STT/TTS systems
4. Your approach for solving latency + barge-in issues
5. Your expected timeline
6. Your preferred architecture improvements for our existing stack
---
# Tech Stack
* React
* Express.js
* Node.js
* Vobiz SIP
* Sarvam AI
* Groq API
* WebSockets
* Real-time audio streaming
Looking for someone who can start immediately and deeply understands real-time conversational AI systems.
# Senior Real-Time Voice AI Engineer Needed (SIP + Streaming STT/TTS + Low Latency + Barge-In)
## Project Overview
We are building a real-time AI voice calling platform for automated outbound/inbound calls that can schedule meetings and handle natural human-like conversations over phone calls.
Current stack:
* Frontend: React
* Backend: Express.js / Node.js
* SIP Provider: Vobiz SIP
* STT/TTS: Sarvam AI
* LLM: Currently Groq API (may migrate later)
* Existing architecture: Custom-built real-time voice pipeline
The system is already functional and capable of making calls, but we need an experienced real-time voice AI engineer to optimize the architecture and solve latency + interruption handling problems.
---
# Current Issues
## 1. High Response Latency (4–7 seconds)
Current flow:
STT → LLM → TTS
Problems:
* AI takes too long to respond
* System waits for full LLM response before TTS
* Audio generation is not properly streamed
* Calls feel unnatural due to response delay
---
## 2. Barge-In / Interruption Problems
We partially implemented interruption handling, but conversation flow still breaks.
Current issues:
* User interrupts AI while it is speaking
* Existing pipeline becomes locked/busy
* Old utterances replay later
* Conversation becomes out-of-sync
* Replay-loop issue:
"Pipeline busy → buffering utterance for replay"
We need proper:
* Cancel-and-restart architecture
* Real-time interruption handling
* Audio buffer flushing
* Immediate TTS stop on user speech
---
## 3. Real-Time Streaming Pipeline Optimization
Need improvements in:
* Streaming STT
* Streaming LLM responses
* Streaming TTS audio
* Turn detection
* Voice Activity Detection (VAD)
* Concurrent session handling
* SIP audio streaming
* Memory cleanup
* Queue management
* Event listener cleanup
---
# Technical Requirements
We are specifically looking for someone with experience in:
* Real-time conversational AI
* Voice calling systems
* SIP/WebRTC architectures
* Streaming audio pipelines
* Low-latency AI systems
* Real-time Node.js systems
* Telephony integrations
* AI interruption/barge-in systems
Strong experience with:
* SIP / RTP
* WebSocket streaming
* Node.js streams
* Real-time media handling
* STT/TTS streaming APIs
* AI voice agents
* Express.js backend optimization
---
# Important Note
We are NOT looking to migrate to LiveKit or rebuild the platform using another framework.
We want to optimize and stabilize our EXISTING custom architecture and make it production-ready.
Please apply only if you are comfortable debugging and improving a custom real-time voice pipeline.
---
# Expected Deliverables
* Reduce response latency to near real-time
* Implement proper interruption/barge-in handling
* Remove replay-loop architecture issues
* Optimize streaming pipeline
* Improve TTS responsiveness
* Improve STT responsiveness
* Proper pipeline cancellation handling
* Stable concurrent call handling
* Production-ready architecture improvements
* Clean/refactored backend implementation
* Technical documentation
---
# Current Technical Problems Observed
Examples from logs:
* "Pipeline busy → buffering utterance for replay"
* Delayed TTS generation
* TTS cancellation issues
* Queued audio replay
* Long LLM wait times
* EventEmitter memory leak warnings
* In-flight request cancellation problems
---
# Ideal Candidate
You have previously worked on:
* AI calling systems
* Voice bots
* SIP-based applications
* Telephony AI
* Real-time audio streaming
* AI assistants with interruption support
* Low-latency media systems
---
# When Applying Please Include
1. Similar voice AI projects you worked on
2. Your experience with SIP/telephony systems
3. Experience with streaming STT/TTS systems
4. Your approach for solving latency + barge-in issues
5. Your expected timeline
6. Your preferred architecture improvements for our existing stack
---
# Tech Stack
* React
* Express.js
* Node.js
* Vobiz SIP
* Sarvam AI
* Groq API
* WebSockets
* Real-time audio streaming
Looking for someone who can start immediately and deeply understands real-time conversational AI systems.