Azure Data Factory Pipeline Development
Budget: ₹600 – ₹1,500 INR
I need a set of Azure Data Factory pipelines to automate data fetching and storage.
Requirements:
- Fetch data for India, US, UK, China, and Russia from the REST API (https://restcountries.com/v3.1/name/{name})
- Save as JSON files with country names as filenames.
- Organize in separate folders for each country in ADLS.
- Trigger to run automatically twice a day at 12:00 AM and 12:00 PM IST.
- Copy customer data from MySQL DB to ADLS if record count > 500.
- If copied, trigger child pipeline to copy product data if customer count > 600.
- Pass customer count to child product pipeline via pipeline parameter.
Ideal Skills & Experience:
- Proficiency in Azure Data Factory.
- Experience with REST APIs.
- Knowledge of Azure Data Lake Storage.
- MySQL database experience.
- Pipeline design and automation expertise.
Requirements:
- Fetch data for India, US, UK, China, and Russia from the REST API (https://restcountries.com/v3.1/name/{name})
- Save as JSON files with country names as filenames.
- Organize in separate folders for each country in ADLS.
- Trigger to run automatically twice a day at 12:00 AM and 12:00 PM IST.
- Copy customer data from MySQL DB to ADLS if record count > 500.
- If copied, trigger child pipeline to copy product data if customer count > 600.
- Pass customer count to child product pipeline via pipeline parameter.
Ideal Skills & Experience:
- Proficiency in Azure Data Factory.
- Experience with REST APIs.
- Knowledge of Azure Data Lake Storage.
- MySQL database experience.
- Pipeline design and automation expertise.
Related categories:
PHP
Data Processing
NoSQL Couch & Mongo
MySQL
Elasticsearch
Data Integration
Data Management
REST API