Azure Data Integration Specialist Needed to do ETL from API to SQL using ADF
Budget: $10 – $100 USD
Job Description
We are looking for an experienced Azure Data Engineer / Data Integration Specialist to design and implement a robust, scalable data pipeline that pulls data from the Blackbaud API and loads it into an Azure SQL Database using Azure Data Factory (ADF).
The goal is to have a fully automated, secure, and monitored ETL pipeline that runs on a scheduled basis and supports future scaling.
Project Scope
1. Data Ingestion
Connect to Blackbaud REST APIs (OAuth authentication)
Handle pagination, rate limits, and API throttling
Extract multiple endpoints (e.g., constituents, gifts, transactions, etc.)
2. Data Transformation
Clean, normalize, and structure raw API JSON
Handle nulls, schema drift, and data type conversions
Add audit fields (load date, source system, batch id)
3. Data Loading
Load data into Azure SQL Database
Support incremental loads (delta logic)
Implement upsert/merge logic where needed
4. Automation & Monitoring
Schedule via Azure Data Factory triggers
Logging, failure alerts, and retry logic
Error handling & performance optimization
This will be 3 pipelines using ADF to ingest using proper architecture in Azure SQL, naming conventions, etc.
We are looking for an experienced Azure Data Engineer / Data Integration Specialist to design and implement a robust, scalable data pipeline that pulls data from the Blackbaud API and loads it into an Azure SQL Database using Azure Data Factory (ADF).
The goal is to have a fully automated, secure, and monitored ETL pipeline that runs on a scheduled basis and supports future scaling.
Project Scope
1. Data Ingestion
Connect to Blackbaud REST APIs (OAuth authentication)
Handle pagination, rate limits, and API throttling
Extract multiple endpoints (e.g., constituents, gifts, transactions, etc.)
2. Data Transformation
Clean, normalize, and structure raw API JSON
Handle nulls, schema drift, and data type conversions
Add audit fields (load date, source system, batch id)
3. Data Loading
Load data into Azure SQL Database
Support incremental loads (delta logic)
Implement upsert/merge logic where needed
4. Automation & Monitoring
Schedule via Azure Data Factory triggers
Logging, failure alerts, and retry logic
Error handling & performance optimization
This will be 3 pipelines using ADF to ingest using proper architecture in Azure SQL, naming conventions, etc.
Related categories:
Data Processing
SQL
Azure
Data Warehousing
OAuth
Data Integration
ETL
API Integration
Data Management
REST API