SQL to BigQuery Data Migration and Automation

I’m looking for a freelancer to help move data from an MS SQL database to Google BigQuery and automate this process using Python scripts. The goal is to centralize the data for better analysis and set up a structure in BigQuery that is organized and scalable for future use. I will provide access to BigQuery so the freelancer can set up datasets and organize the data directly within the platform.

Scope of Work:

Data Extraction and Transformation:
- Extract data from SQL tables (e.g., inventory, item details, sales history) based on specific fields and filters
- Ensure data is cleaned and formatted for smooth loading into BigQuery.

BigQuery Setup and Organization:
- Access BigQuery to set up and structure datasets (e.g., inventory_data, sales_data, item_details).
-Ensure the datasets are organized and optimized for analytics.

Python Automation:
Develop Python scripts to:
- Extract data from the SQL database.
- Load data into BigQuery, ensuring incremental updates and handling duplicates.
- Schedule the scripts to run multiple times a day using a task scheduler (e.g., cron or similar).

Error Handling and Logging:
- Add error-handling mechanisms in the scripts to handle failed transfers and log success/error details for every execution.

Documentation and Guidance:
- Provide clear instructions for running and maintaining the scripts.
- Minimal guidance to help me perform small tasks like testing queries or troubleshooting basic issues in BigQuery if needed.

Requirements:
- Strong experience with SQL and Python.
- Proficiency in Google BigQuery for setting up and managing datasets.
- Familiarity with ETL/ELT processes and cloud platforms like Google Cloud.


Deliverables:
Python scripts that:
- Extract data from SQL and load it into BigQuery.
- Run automatically at scheduled intervals.
- Organized BigQuery datasets ready for analysis.
- Documentation for maintaining the process and handling common issues.