ETL pipeline w/ Terraform, Apache Airflow and Kubernetes
Budget: £20 – £250 GBP
A basic ETL pipeline that ingests data from two MySQL databases periodically is needed. The workflow should be managed by Terraform and orchestrated by Apache Airflow to run Spark jobs that perform basic transformations on the ingested data within a Kubernetes cluster, before writing the transformed data to a local SQLite database or file.
The code should be written in Python and the deliverable should be a testable repository containing the scripts and configs. Key technologies: Python, Airflow, Kubernetes and Terraform.
The code should be written in Python and the deliverable should be a testable repository containing the scripts and configs. Key technologies: Python, Airflow, Kubernetes and Terraform.