ADF ETL Pipeline Development
Budget: ₹600 – ₹1,500 INR
I need an expert to design and implement a data transformation pipeline in Azure Data Factory (ADF). This pipeline will extract and transform raw transaction data from Azure Blob Storage (in CSV format) into enriched features using SQL, with scheduled triggers.
Key Requirements:
- Create ADF pipelines to ingest raw data from Azure Blob Storage to Data Lake.
- Use SQL queries to:
- Clean data: Handle null values, remove duplicates.
- Generate basic features: Transaction amount buckets, login frequency, and time-based flag
- Configure tumbling window triggers for periodic batch processing.
- Set up event-based triggers for real-time ingestion scenarios.
- Include comprehensive documentation for:
- Pipeline design
- SQL logic
- Trigger configuration
Ideal Skills & Experience:
- Strong experience with Azure Data Factory
- Proficiency in SQL
- Familiarity with Azure Blob Storage and Data Lake
- Experience in data transformation and pipeline development
- Good documentation skills
Looking for a detail-oriented professional who can ensure a robust and well-documented ETL process.
Key Requirements:
- Create ADF pipelines to ingest raw data from Azure Blob Storage to Data Lake.
- Use SQL queries to:
- Clean data: Handle null values, remove duplicates.
- Generate basic features: Transaction amount buckets, login frequency, and time-based flag
- Configure tumbling window triggers for periodic batch processing.
- Set up event-based triggers for real-time ingestion scenarios.
- Include comprehensive documentation for:
- Pipeline design
- SQL logic
- Trigger configuration
Ideal Skills & Experience:
- Strong experience with Azure Data Factory
- Proficiency in SQL
- Familiarity with Azure Blob Storage and Data Lake
- Experience in data transformation and pipeline development
- Good documentation skills
Looking for a detail-oriented professional who can ensure a robust and well-documented ETL process.