GCP ETL Pipeline Development

I need an end-to-end ETL pipeline that moves data from our on-premise databases into Google Cloud, transforms it, and lands it cleanly in BigQuery. The core stack must be Airflow for orchestration, Dataproc (running PySpark) for heavy transformations, and native BigQuery SQL for final modelling and reporting layers.

You will design and implement:
• Secure ingestion from the on-prem source into GCS staging
• Airflow DAGs that trigger Dataproc jobs, handle retries, logging and alerting
• PySpark transformation scripts on Dataproc, tuned for performance and cost
• BigQuery SQL models that expose the refined tables
• Parameterised configuration so environments can be promoted from dev to prod without code changes

Acceptance criteria:
– A scheduled Airflow DAG runs end-to-end without manual intervention.
– Data in BigQuery matches source row counts and key metrics for at least three historical loads.
– Code, requirements, and deployment instructions are delivered in a Git-ready repository.

If you have prior experience marrying Airflow, Dataproc and BigQuery at scale, I’m ready to start right away.