Databricks Spot Cost Optimization
Budget: ₹600 – ₹1,500 INR
I need an Azure Databricks specialist to re-architect my small batch-processing cluster (1-10 nodes) so it reliably runs on spot instances and trims our monthly bill. The objective is simple: reduce overall operational costs without sacrificing job completion time or data integrity.
What I expect from you:
• Analyse the current workspace, jobs and autoscaling behaviour.
• Design and apply an optimal spot-instance strategy (instance mix, bid price, fault-tolerant settings, pool configuration, autoscaling rules).
• Implement job-level tweaks such as task parallelism, cluster reuse and termination timing to prevent idle burn.
• Set up cost-tracking dashboards or alerts so I can see savings in real time.
• Deliver a short hand-off guide that explains the new configuration and roll-back steps.
Acceptance criteria
• Batch jobs finish within their existing SLA.
• Spot utilisation ≥ 80 % over a representative 7-day window.
• Demonstrated cost reduction of at least 30 % compared with the current on-demand setup.
Tools you are likely to touch: Azure Portal, Databricks CLI, Terraform or ARM templates (if you prefer IaC), and Azure Monitor/Cost Management. I’ll grant temporary contributor access once we agree on the approach, and I’m ready to start as soon as you are.
What I expect from you:
• Analyse the current workspace, jobs and autoscaling behaviour.
• Design and apply an optimal spot-instance strategy (instance mix, bid price, fault-tolerant settings, pool configuration, autoscaling rules).
• Implement job-level tweaks such as task parallelism, cluster reuse and termination timing to prevent idle burn.
• Set up cost-tracking dashboards or alerts so I can see savings in real time.
• Deliver a short hand-off guide that explains the new configuration and roll-back steps.
Acceptance criteria
• Batch jobs finish within their existing SLA.
• Spot utilisation ≥ 80 % over a representative 7-day window.
• Demonstrated cost reduction of at least 30 % compared with the current on-demand setup.
Tools you are likely to touch: Azure Portal, Databricks CLI, Terraform or ARM templates (if you prefer IaC), and Azure Monitor/Cost Management. I’ll grant temporary contributor access once we agree on the approach, and I’m ready to start as soon as you are.