Automate Databricks Pipelines via Terraform

I’m ready to move our Azure Databricks workloads to an infrastructure-as-code model so every part of the data pipeline can be recreated or updated with a single Terraform apply. The objective is full pipeline automation, driven on a fixed schedule, and wired into three data sources that are already live:

• Azure SQL Database
• Azure Data Lake
• Azure Cosmos DB

Here’s what I need from you:

– Terraform code (HCL) that builds the Databricks workspace, clusters, jobs/notebooks, and any supporting resources required for those sources.
– Scheduled execution logic defined in Terraform so the pipeline runs automatically at the specified times without manual triggers.
– Clear variables, state-handling strategy, and README-style documentation that lets my team replicate the deployment in another subscription with minimal changes.
– A brief walkthrough call or screencast showing the workflow from terraform init to a successful scheduled run that pulls data from all three sources.

If you have previous experience linking these exact Azure services through Databricks and Terraform, I’d love to see it. Let’s make the pipeline reproducible, version-controlled, and hassle-free.