LLM Capacity Planning Analysis
Budget: $30 – $250 USD
I’m trying to estimate our system capacity for long-context LLM calls, based on our alert workload.
Assumptions per message (per alert):
Avg input tokens: 34K
Avg output tokens: 5K
GPU setups to compare:
8× RTX 6000 Ada (48GB) = 384GB total
2× RTX 6000 Ada (48GB) = 96GB total
4× H200 (141GB) = 564GB total
Models to compare on each setup:
DeepSeek v3.2
GPT-OSS 20B
GPT-OSS 120B
For each (GPU setup × model), I want three outputs:
Concurrent messages (how many requests we can serve “at the same time”)
Time per message (latency in seconds/minutes per request)
Total messages in 10 hours (throughput/capacity over a 10-hour window)
The goal is to produce a clear comparison table that shows capacity + latency across these options.
Thanks!
Assumptions per message (per alert):
Avg input tokens: 34K
Avg output tokens: 5K
GPU setups to compare:
8× RTX 6000 Ada (48GB) = 384GB total
2× RTX 6000 Ada (48GB) = 96GB total
4× H200 (141GB) = 564GB total
Models to compare on each setup:
DeepSeek v3.2
GPT-OSS 20B
GPT-OSS 120B
For each (GPU setup × model), I want three outputs:
Concurrent messages (how many requests we can serve “at the same time”)
Time per message (latency in seconds/minutes per request)
Total messages in 10 hours (throughput/capacity over a 10-hour window)
The goal is to produce a clear comparison table that shows capacity + latency across these options.
Thanks!