Databricks Job Runtime Optimization - 26/05/2026 04:44 EDT
Budget: ₹1,500 – ₹12,500 INR
Our nightly Databricks notebook that ingests structured data from Azure Data Lake has crept from a consistent 3-hour finish to nearly 8 hours. The slowdown is clearly happening during the data-reading phase; once the tables are in memory the transformations flow as expected.
I need you to dig into the cluster and the code, pinpoint the I/O bottlenecks, and implement the fixes that will bring the total wall-clock time back to (or better than) the original 3 hours. Spark partitioning strategy, file format selection, caching, Delta Lake optimisation, cluster sizing, Auto Loader configuration—whatever combination gets us there is fair game as long as it remains maintainable and cost-conscious on Azure.
Deliverables
• Revised notebook (or modular scripts) with all performance-related changes clearly commented
• Brief before-and-after report showing key metrics: input rows per second, stage/task durations, and total runtime
• Short hand-off note outlining any cluster-level settings or schedule tweaks applied
Acceptance criterion: a full test run on my workspace that completes in ≤ 3 hours while producing identical outputs.
I need you to dig into the cluster and the code, pinpoint the I/O bottlenecks, and implement the fixes that will bring the total wall-clock time back to (or better than) the original 3 hours. Spark partitioning strategy, file format selection, caching, Delta Lake optimisation, cluster sizing, Auto Loader configuration—whatever combination gets us there is fair game as long as it remains maintainable and cost-conscious on Azure.
Deliverables
• Revised notebook (or modular scripts) with all performance-related changes clearly commented
• Brief before-and-after report showing key metrics: input rows per second, stage/task durations, and total runtime
• Short hand-off note outlining any cluster-level settings or schedule tweaks applied
Acceptance criterion: a full test run on my workspace that completes in ≤ 3 hours while producing identical outputs.