Restructuring UrbanChat AI Voice Platform

About UrbanChat
UrbanChat is a scalable AI voice agent platform that allows businesses to sign up and build their own AI voice agents to handle customer calls. It works similarly to Retell AI, providing a fully automated AI-powered call-handling solution.

Our current setup integrates Twilio for telephony, ElevenLabs for Text-to-Speech TTS, OpenAI Realtime 4o Mini for Speech-to-Text STT and LLM Natural Language Processing.

We want to restructure our architecture to be highly modular and flexible, allowing us to easily integrate multiple providers for each component without modifying the core backend.

Project Overview
Currently, our AI voice agent relies on:
✅ Telephony – Twilio for making and receiving calls
✅ TTS – ElevenLabs for AI-generated speech
✅ STT – OpenAI Realtime for transcribing user speech
✅ LLM – OpenAI for conversation intelligence

We want to decouple each of these components into separate modular frameworks, making it easy to integrate alternative providers without affecting other parts of the system.

Goals of This Project
Build a modular and scalable framework for Telephony, TTS, STT, and LLM.
Ensure that new providers can be added or swapped with minimal backend changes.
Maintain latency under 1000ms for real-time conversations.
Work on the existing codebase and refactor where needed.
Prepare comprehensive documentation including APIs and integration guidelines.
Your Role
You will be responsible for designing and implementing a modular architecture for the AI voice agent. This will involve breaking down the system into four independent frameworks:

1. Telephony Framework
Currently using Twilio, but should support other providers like Vonage, Exotel, etc.
Abstract call-handling logic so that switching providers requires minimal code changes.
Maintain lowest-latency communication between telephony and the AI engine.
Ensure compatibility with SIP, WebRTC, and standard telephony APIs.
Provide APIs for call initiation, phone number purchase, call handling, and termination.
From the backend, if we need to add another service provider, we should be able to integrate their API, and their phone numbers will be available for purchase on the frontend. This means customers can buy numbers from different providers directly through our platform.

2 TTS Text-to-Speech Framework
Currently using ElevenLabs, but should support Sarvam AI, Deepgram, Google TTS, etc.
Implement a standardized API for different TTS providers.
Ensure real-time streaming support for seamless voice interactions.
Optimize for low-latency response generation below 1000ms.
Provide fallback mechanisms in case a provider fails.

3 STT Speech-to-Text Framework
Currently using OpenAI Realtime, but should support Deepgram, Whisper, Google STT, etc.
Implement a plug-and-play interface for STT providers.
Maintain high transcription accuracy and low latency.
Support real-time streaming transcription for conversational AI.
Optimize handling of different accents, noise levels, and speech variations.

4 LLM AI Engine Framework
Currently using OpenAI GPT-4o Mini, but should support Claude, Gemini, Mistral, etc.
Design an API wrapper that allows switching between different AI models with minimal effort.
Optimize response generation for low latency and high conversation quality.
Support context-aware memory and multi-turn conversations.
Implement a fallback system in case of API failures or rate limits.

Expectations and Deliverables
✅ Modular Architecture
Each framework should be built as an independent module with a standardized API, so we can integrate any provider in the future with minimal changes.

✅ Real-Time Optimization
The entire system must maintain end-to-end latency under 1000ms, ensuring a smooth conversation experience.

✅ Code Refactoring and Integration
You will be working on our existing codebase, improving structure and modularity without breaking functionality.

✅ Comprehensive Documentation

Full API documentation for each framework.
Step-by-step guides for adding new providers.
Code structure explanations for future developers.
Who We Are Looking For
? Experience with TTS, STT, LLM, and Telephony APIs Twilio, ElevenLabs, OpenAI, Deepgram, Sarvam AI, etc.
? Strong expertise in building modular and scalable architectures
? Real-time speech processing and low-latency optimization
? Familiarity with Node.js, Python, WebSockets, and cloud services
? Prior experience in conversational AI or AI voice platforms is a huge plus

This is a long-term role, and we will assign work module by module rather than giving everything at once. If you deliver high-quality work, we may extend your engagement to additional AI-related projects.