Cost-Aware Scaling in Multi-User LLM Systems
Budget: ₹600 – ₹1,500 INR
Idea in Simple Words
Instead of only checking speed/errors, you also check money.
LLMs are expensive, so the project looks at performance + cost together.
Goal
• Find how to keep apps both scalable and affordable.
What You’ll Measure
• Response times under load
• Error rates under load
• Cost per 1000 requests (API billing / GPU runtime)
• Trade-off → faster = more costly, slower = cheaper
Differentiation from Original
Original = focuses on speed, errors, and resources.
This = adds cost as a first-class metric and explores cost vs performance tradeoffs.
What You’ll Do
1. Simulate multiple users on an LLM API (e.g., OpenAI, HuggingFace).
2. Measure total API token usage cost at different user loads.
3. Deploy a local model and calculate GPU/CPU running cost.
4. Suggest cost-aware strategies:
• Route small/simple queries to a smaller model.
• Only use a big model for complex queries.
• Auto-scale GPUs only when traffic spikes.
Instead of only checking speed/errors, you also check money.
LLMs are expensive, so the project looks at performance + cost together.
Goal
• Find how to keep apps both scalable and affordable.
What You’ll Measure
• Response times under load
• Error rates under load
• Cost per 1000 requests (API billing / GPU runtime)
• Trade-off → faster = more costly, slower = cheaper
Differentiation from Original
Original = focuses on speed, errors, and resources.
This = adds cost as a first-class metric and explores cost vs performance tradeoffs.
What You’ll Do
1. Simulate multiple users on an LLM API (e.g., OpenAI, HuggingFace).
2. Measure total API token usage cost at different user loads.
3. Deploy a local model and calculate GPU/CPU running cost.
4. Suggest cost-aware strategies:
• Route small/simple queries to a smaller model.
• Only use a big model for complex queries.
• Auto-scale GPUs only when traffic spikes.