Resolve Databricks Processing Crash

I’m running a real-time, production workload on Azure Databricks that ingests structured data from CSV/SQL sources and processes it in Spark notebooks. The jobs run fine until the transformation stage, where they suddenly throw runtime errors and the cluster crashes. I need someone who can jump in quickly, reproduce the failure, trace the exact root cause, and deliver a rock-solid fix without disrupting the live data feed.

You should be fully comfortable with Databricks Runtime, Spark (Scala or PySpark), Delta Lake tables, cluster configuration, and Azure storage connectors, as that’s the stack in play. If a configuration tweak, code refactor, or resource-scaling change is required, implement it, document the change, and prove stability with reruns under production-level load.

Deliverables (all required):
• Root-cause analysis summary with error logs highlighted
• Updated notebook / job code or cluster settings that eliminate the crash
• Validation evidence: successful processing runs and performance baseline post-fix

Once these items pass review in the production workspace, the project is complete.