High Availability GPU Inference API using Kubernetes
Budget: $30 – $250 USD
To apply for this project, you need to have industry experience in ML operation, specifically using Kubernetes in LLM model inference on multiple GPUs/nodes!!!! In such case, you should complete this easily in 1-2 days
I'm looking for a skilled developer who can set up a Kubernetes deployment that supports multiple identical GPU inference APIs with high availability. Environment: Ubuntu 20, Docker2+, Python, Pytorch.
I have a Nvidia Docker image, currently its docker container can serve a llama model inference on a full nvidia A6000 GPU using fastAPI mapping to a local port #. works perfectly fine for all model inference jobs on batches...
I can start multiple docker containers using same docker image pointing to different fast API port. just have to provide a new A6000 nvidia GPU for a new docker container.
Goal:
Now I want to use Kubernetes to manage those model serving APIs (http/ports):
1. to route any batch needs or dynamic user request to Kubernetes, to pickup local free resources
2. to speed up model inference batch process N times faster
3. to monitor each GPU, node situations and status
I am open to any design suggestion, prefer industry ML operation best practice
In the end, I need to spped up llama model inference speed, reliable, and scalable, and, Kubernetes can be controled and monitored...
Key Points:
- Develop a GPU inference API using Python.
- Set up Kubernetes deployment with high availability.
- Ensure the APIs are running on multiple identical GPUs.
Ideal Skills:
- Proficiency in Python for GPU inference API development.
- Experience with Kubernetes for high availability deployments.
- Understanding of GPU infrastructure and coordination.
The data storage for the APIs should be handled on a local server, so experience with setting up data storage on a local server is a plus. Your responsibility will be to ensure that the system can scale as needed, while maintaining high availability using Kubernetes.
I'm looking for a skilled developer who can set up a Kubernetes deployment that supports multiple identical GPU inference APIs with high availability. Environment: Ubuntu 20, Docker2+, Python, Pytorch.
I have a Nvidia Docker image, currently its docker container can serve a llama model inference on a full nvidia A6000 GPU using fastAPI mapping to a local port #. works perfectly fine for all model inference jobs on batches...
I can start multiple docker containers using same docker image pointing to different fast API port. just have to provide a new A6000 nvidia GPU for a new docker container.
Goal:
Now I want to use Kubernetes to manage those model serving APIs (http/ports):
1. to route any batch needs or dynamic user request to Kubernetes, to pickup local free resources
2. to speed up model inference batch process N times faster
3. to monitor each GPU, node situations and status
I am open to any design suggestion, prefer industry ML operation best practice
In the end, I need to spped up llama model inference speed, reliable, and scalable, and, Kubernetes can be controled and monitored...
Key Points:
- Develop a GPU inference API using Python.
- Set up Kubernetes deployment with high availability.
- Ensure the APIs are running on multiple identical GPUs.
Ideal Skills:
- Proficiency in Python for GPU inference API development.
- Experience with Kubernetes for high availability deployments.
- Understanding of GPU infrastructure and coordination.
The data storage for the APIs should be handled on a local server, so experience with setting up data storage on a local server is a plus. Your responsibility will be to ensure that the system can scale as needed, while maintaining high availability using Kubernetes.