Azure Data Engineering Expertise Needed
Budget: ₹750 – ₹1,250 INR
I need an experienced Data Engineer (around five years in the field) to strengthen our Azure-based analytics environment. The immediate focus is threefold: migrate existing datasets, design a robust warehouse layer, and build automated ETL pipelines that keep everything in sync.
Our data arrives from several places—traditional relational databases, and a series of partner APIs—so you’ll have to feel comfortable normalising and orchestrating very different payloads. Day-to-day work will rely heavily on Azure Data Factory for orchestration, Azure SQL Database for storage, and Azure Databricks for any large-scale transformations that go beyond standard activities.
Deliverables I expect:
• A detailed migration plan with rollback steps
• Production-ready ETL pipelines in ADF, parameterised and linked to Dev/Test/Prod environments
• Optimised warehouse schema deployed in Azure SQL Database and documented for our BI team
• Databricks notebooks (or jobs) that handle complex transformations, checked into our repo and scheduled through ADF
• A concise runbook and architecture diagram so future engineers can maintain and extend the setup
Completion means the pipelines run on a schedule without failure for at least one full cycle, data integrity checks come back clean, and the solution is handed over with clear documentation and version-controlled code. If this matches your skill set, I’m eager to get started quickly.
Our data arrives from several places—traditional relational databases, and a series of partner APIs—so you’ll have to feel comfortable normalising and orchestrating very different payloads. Day-to-day work will rely heavily on Azure Data Factory for orchestration, Azure SQL Database for storage, and Azure Databricks for any large-scale transformations that go beyond standard activities.
Deliverables I expect:
• A detailed migration plan with rollback steps
• Production-ready ETL pipelines in ADF, parameterised and linked to Dev/Test/Prod environments
• Optimised warehouse schema deployed in Azure SQL Database and documented for our BI team
• Databricks notebooks (or jobs) that handle complex transformations, checked into our repo and scheduled through ADF
• A concise runbook and architecture diagram so future engineers can maintain and extend the setup
Completion means the pipelines run on a schedule without failure for at least one full cycle, data integrity checks come back clean, and the solution is handed over with clear documentation and version-controlled code. If this matches your skill set, I’m eager to get started quickly.