Cloud & Data Engineer: GCP, AWS, Airflow
Budget: $30 – $250 USD
I'm seeking an expert in setting up Google Cloud, AWS EMR, and Apache Airflow. Your responsibilities will include:
- Setting up infrastructure: You'll need to build a robust and scalable cloud infrastructure tailored to our needs.
- Writing custom scripts: This will involve scripting in Python to meet our specific data handling requirements.
- Data processing workflows: You will design and implement efficient data processing workflows using Airflow.
Main project is to create views from google bigquery dataset and read it in aws emr using Workload identity federation ,create airflow jobs and then dump it to sql server.
Shall be expert in terraform scripting as well
Your expertise in these areas will be critical to the success of this project. Prior experience with large-scale data processing and cloud infrastructure is highly desirable. Please ensure your bid reflects your level of expertise and experience in these areas.
- Setting up infrastructure: You'll need to build a robust and scalable cloud infrastructure tailored to our needs.
- Writing custom scripts: This will involve scripting in Python to meet our specific data handling requirements.
- Data processing workflows: You will design and implement efficient data processing workflows using Airflow.
Main project is to create views from google bigquery dataset and read it in aws emr using Workload identity federation ,create airflow jobs and then dump it to sql server.
Shall be expert in terraform scripting as well
Your expertise in these areas will be critical to the success of this project. Prior experience with large-scale data processing and cloud infrastructure is highly desirable. Please ensure your bid reflects your level of expertise and experience in these areas.