Azure Databricks Structured Streaming
Budget: $250 – $750 USD
I'm a Data Engineer and new to Azure.
Our current architecture is designed to source data incrementally from On-prem relational (SQL Server designed in 3rd Normal Form) tables and load into Azure Data Lake Store Gen 1 (Bronze Layer). Once the data is loaded in the Bronze layer, we use Azure Databricks to massgae/apply business rules by joining tables in Bronze Layer and load into Silver/Gold layer. Because the data is in 3NF form, we look for changes in any of the source tables and get the related information from the other tables (by Joining) and load the Silver/Gold layer. All of this is done on a nightly schedule.
Now we have a need to have the data to be available more real-time. So we're planning on enabling Change Data Capture on the On-prem source and capture all the transactions that happen on the database and send it to an Azure Event Hub. We need to source the data from the Azure Event Hub and process the data using Azure Databricks Structured Streaming and load the Silver/Gold layer tables.
I'm looking for someone that can help me achieve this by taking a simple use case (at-least with the Azure Databricks Structured Streaming part). Please reach out to me if you have any questions.
Our current architecture is designed to source data incrementally from On-prem relational (SQL Server designed in 3rd Normal Form) tables and load into Azure Data Lake Store Gen 1 (Bronze Layer). Once the data is loaded in the Bronze layer, we use Azure Databricks to massgae/apply business rules by joining tables in Bronze Layer and load into Silver/Gold layer. Because the data is in 3NF form, we look for changes in any of the source tables and get the related information from the other tables (by Joining) and load the Silver/Gold layer. All of this is done on a nightly schedule.
Now we have a need to have the data to be available more real-time. So we're planning on enabling Change Data Capture on the On-prem source and capture all the transactions that happen on the database and send it to an Azure Event Hub. We need to source the data from the Azure Event Hub and process the data using Azure Databricks Structured Streaming and load the Silver/Gold layer tables.
I'm looking for someone that can help me achieve this by taking a simple use case (at-least with the Azure Databricks Structured Streaming part). Please reach out to me if you have any questions.