Azure Database Setup & Data Pipeline Creation
Budget: £250 – £750 GBP
Freelancer Task: Azure MS SQL Database & Data Pipeline Setup
Project Scope
The freelancer will deploy a cloud-based Microsoft Azure SQL Database, set up an automated data pipeline for Excel uploads, and provide a management interface for querying the data.
Requirements
1. Azure SQL Database Setup
Deploy an Azure SQL Database (PaaS) instance.
Create three tables using data from provided Excel sheets.
Define appropriate data types, constraints, and indexing for optimised performance.
Provide SQL scripts for table creation and initial data import.
2. Azure Virtual Machine (VM) for Management
Provision an Azure Virtual Machine (Windows-based) to serve as a management server.
Install and configure Microsoft SQL Server Management Studio (SSMS) for querying the database.
Ensure secure remote access via RDP with proper firewall rules.
3. Data Pipeline with Azure Data Factory (ADF)
Set up an Azure Data Factory (ADF) pipeline to upload and process new Excel versions.
Configure the pipeline to support incremental or full data updates as needed.
Implement error handling, logging, and monitoring using Azure Monitor and Log Analytics.
4. Cloud Storage for Excel Uploads
Set up an Azure Blob Storage container (or appropriate storage option) as the designated location for Excel files.
Ensure automated pickup of new files by ADF.
Show how to upload new excel files.
Deliverables
Azure SQL Database with three tables and initial data import.
Azure Virtual Machine with SQL Server Management Studio (SSMS) installed. (Connected to database)
Automated Data Pipeline in Azure Data Factory (ADF).
Azure Blob Storage setup for Excel file uploads. (Or better option)
Project Scope
The freelancer will deploy a cloud-based Microsoft Azure SQL Database, set up an automated data pipeline for Excel uploads, and provide a management interface for querying the data.
Requirements
1. Azure SQL Database Setup
Deploy an Azure SQL Database (PaaS) instance.
Create three tables using data from provided Excel sheets.
Define appropriate data types, constraints, and indexing for optimised performance.
Provide SQL scripts for table creation and initial data import.
2. Azure Virtual Machine (VM) for Management
Provision an Azure Virtual Machine (Windows-based) to serve as a management server.
Install and configure Microsoft SQL Server Management Studio (SSMS) for querying the database.
Ensure secure remote access via RDP with proper firewall rules.
3. Data Pipeline with Azure Data Factory (ADF)
Set up an Azure Data Factory (ADF) pipeline to upload and process new Excel versions.
Configure the pipeline to support incremental or full data updates as needed.
Implement error handling, logging, and monitoring using Azure Monitor and Log Analytics.
4. Cloud Storage for Excel Uploads
Set up an Azure Blob Storage container (or appropriate storage option) as the designated location for Excel files.
Ensure automated pickup of new files by ADF.
Show how to upload new excel files.
Deliverables
Azure SQL Database with three tables and initial data import.
Azure Virtual Machine with SQL Server Management Studio (SSMS) installed. (Connected to database)
Automated Data Pipeline in Azure Data Factory (ADF).
Azure Blob Storage setup for Excel file uploads. (Or better option)