Deploying PyTorch+Gradio endpoint onto Azure-based scalable system, probably a AKS cluster (open to better ideas)
Budget: $250 – $750 USD
I have a PyTorch + Gradio codebase that I want to deploy on Azure, with efficient autoscaling. The codebase is for audio inferencing. My assumption is Azure Kubernetes would be best and most cost efficient, but I'm open to other ideas.
My attempt was to use a load balancer that connects to GPU-powered instances that are auto scaled up and down based on average of connections to the individual instances.
SPECIFICS:
- PyTorch 2.0.1 and Python 3.8 are required for this codebase, along with the codebase's own requirements.txt installation. I'm open to Docker usage, but in my own testing, PyTorch with GPU support is quite large and because of that, docker image downloading and extractions slows down the start time of an instance very significantly.
- I would like to use spot instances as much as possible, and only fall back to an on-demand instance if there's no spot instances. Ideally, if there is an on-demand instance being used, i'd like to attempt to replace it using a spot instance when a spot instance becomes available.
- Since this is an inferencing endpoint, I need minimal delay with scaling out, so this is an area where I really need the proper expertise.
If things go well with us, I would definitely have additional work I would need help with.
DELIVERABLES: I need to be able to configure and run this myself, and adjust things, therefore I would want to be given Azure CLI commands and config files that I could use, along with any instructions to properly configure identities, permissions, etc.
My attempt was to use a load balancer that connects to GPU-powered instances that are auto scaled up and down based on average of connections to the individual instances.
SPECIFICS:
- PyTorch 2.0.1 and Python 3.8 are required for this codebase, along with the codebase's own requirements.txt installation. I'm open to Docker usage, but in my own testing, PyTorch with GPU support is quite large and because of that, docker image downloading and extractions slows down the start time of an instance very significantly.
- I would like to use spot instances as much as possible, and only fall back to an on-demand instance if there's no spot instances. Ideally, if there is an on-demand instance being used, i'd like to attempt to replace it using a spot instance when a spot instance becomes available.
- Since this is an inferencing endpoint, I need minimal delay with scaling out, so this is an area where I really need the proper expertise.
If things go well with us, I would definitely have additional work I would need help with.
DELIVERABLES: I need to be able to configure and run this myself, and adjust things, therefore I would want to be given Azure CLI commands and config files that I could use, along with any instructions to properly configure identities, permissions, etc.