Scalable Azure ETL Pipeline

I need an end-to-end data pipeline built with Azure Data Factory orchestrating Azure Databricks notebooks. The solution must ingest data that mainly arrives as CSV files, JSON files, SQL-database extracts, plus occasional Parquet or existing Delta tables, and land everything in Azure Data Lake Storage Gen2.

Inside Databricks, you will use PySpark and SQL to profile, clean, validate, and transform the raw feeds, applying robust error handling so that bad records are quarantined without breaking the run. Incremental (CDC) logic is required so that subsequent loads update or append only what has changed. Processed data should be written back in Delta Lake format (Parquet is an acceptable fallback where Delta is not supported) and properly partitioned for query performance.

The pipeline must be:

• Automated – triggered on schedule or event, with parameterised ADF pipelines and Databricks job clusters.
• Optimised – tuned cluster sizing, caching, and code refactoring to keep runtimes and costs down.
• Tested – unit tests for transformations and a validation step that confirms row counts and schema before promotion.
• CI/CD-ready – ARM/Bicep or Terraform templates plus YAML or JSON release definitions so I can deploy to dev, test, and prod with a single click.
• Documented – architecture diagram, run-book, and inline notebook comments explaining each step.

On delivery I expect:

1. All ADF JSON, Databricks notebooks (.dbc or .py), and infrastructure-as-code scripts checked into a Git repository.
2. A short walkthrough video or live demo showing an initial full load followed by an incremental load and rollback test.
3. A deployment guide and a troubleshooting FAQ.

If you have production experience with ADLS Gen2, Delta Lake, and PySpark and can commit to a quick, well-documented turnaround, I look forward to collaborating.