Python SQL Data Engineer
Budget: $250 – $750 NZD
I am looking for a skilled Python and SQL Data Engineer to assist with various tasks related to structured data.
Ideal Skills:
- Proficiency in Python and SQL is a must.
- Previous experience in data extraction, transformation, and loading.
I have 1000 large tables in the source SQL Server database where DML operations are performed. We have a nightly job that processes these tables, but it currently fails to capture deletes from the source tables. The destination is a Snowflake database. However, I face several constraints:
a) I cannot make any changes to the source tables.
b) I cannot use Change Data Capture (CDC).
c) A full table refresh is not feasible due to the large size of the source tables.
Given these constraints, what solutions would you recommend to ensure deletes are accurately reflected in the Snowflake database?"
Ideal Skills:
- Proficiency in Python and SQL is a must.
- Previous experience in data extraction, transformation, and loading.
I have 1000 large tables in the source SQL Server database where DML operations are performed. We have a nightly job that processes these tables, but it currently fails to capture deletes from the source tables. The destination is a Snowflake database. However, I face several constraints:
a) I cannot make any changes to the source tables.
b) I cannot use Change Data Capture (CDC).
c) A full table refresh is not feasible due to the large size of the source tables.
Given these constraints, what solutions would you recommend to ensure deletes are accurately reflected in the Snowflake database?"