Cloud Deployment: Airflow & Spark POC
Budget: ₹1,500 – ₹12,500 INR
I'm looking for an expert to help me deploy Airflow and Spark in the cloud, specifically on Azure using the Azure Kubernetes Service (AKS). The project involves creating a Directed Acyclic Graph (DAG) in Airflow that triggers a Spark cluster, runs a Pi calculation job, and subsequently closes the cluster upon completion.
we'd like to deploy the same on azure kubernetes service. On AKS, the Spark job also needs to read data from ADLS Gen2 and write to ADLS Gen2 in hudi/deltalake/iceberg.
The ideal freelancer for this project should have:
- Extensive experience with Apache Airflow and Apache Spark.
- Proficient knowledge of Azure and its services, particularly the Azure Kubernetes Service (AKS).
- Skills in creating Airflow DAGs.
This is a proof of concept (POC) project with potential for further collaboration if successful.
we'd like to deploy the same on azure kubernetes service. On AKS, the Spark job also needs to read data from ADLS Gen2 and write to ADLS Gen2 in hudi/deltalake/iceberg.
The ideal freelancer for this project should have:
- Extensive experience with Apache Airflow and Apache Spark.
- Proficient knowledge of Azure and its services, particularly the Azure Kubernetes Service (AKS).
- Skills in creating Airflow DAGs.
This is a proof of concept (POC) project with potential for further collaboration if successful.