Azure Data Engineer Needed for ADF Enhancement
Budget: ₹600 – ₹1,000 INR
I'm seeking a seasoned Azure Data Engineer to augment our current Azure Data Factory (ADF) pipeline. This pipeline is responsible for processing data from an Azure SQL Database and adheres to a medallion architecture (Landing -> Bronze -> Silver -> Gold).
Key Tasks:
- Extend the existing pipeline to accommodate file-based ingestion from Azure Data Lake Storage (ADLS).
- Implement a SQL-based metadata store for monitoring pipeline execution.
Data Formats: The new pipeline extensions will need to handle ingestion of multiple data formats from ADLS, specifically CSV, Parquet, and JSON.
Ideal Skills:
- Extensive experience with Azure Data Factory (ADF).
- Proficient in working with Azure Data Lake Storage (ADLS).
- Strong understanding of medallion architecture.
- Capable of implementing SQL-based metadata stores.
- Competent in handling multiple data formats including CSV, Parquet, and JSON.
- Experience with schema evolution management, specifically with automatic schema inference.
- Proficient in managing and querying Azure SQL Database.
Please note that the new pipeline extensions do not require any data transformations as of now.
The project should be completed within 1 day.
Key Tasks:
- Extend the existing pipeline to accommodate file-based ingestion from Azure Data Lake Storage (ADLS).
- Implement a SQL-based metadata store for monitoring pipeline execution.
Data Formats: The new pipeline extensions will need to handle ingestion of multiple data formats from ADLS, specifically CSV, Parquet, and JSON.
Ideal Skills:
- Extensive experience with Azure Data Factory (ADF).
- Proficient in working with Azure Data Lake Storage (ADLS).
- Strong understanding of medallion architecture.
- Capable of implementing SQL-based metadata stores.
- Competent in handling multiple data formats including CSV, Parquet, and JSON.
- Experience with schema evolution management, specifically with automatic schema inference.
- Proficient in managing and querying Azure SQL Database.
Please note that the new pipeline extensions do not require any data transformations as of now.
The project should be completed within 1 day.