Python Data Cleaning Pipeline
Budget: $10 – $100 USD
I have a production database full of raw numerical records that first need to be cleaned and processed before any modelling work can begin. The task centres on building a repeatable Python pipeline that pulls the data directly from the database, applies rigorous quality checks, and outputs a tidy, analysis-ready table.
I expect the script to connect via SQL (SQLAlchemy or similar), ingest the relevant tables, and then handle missing values, outliers, type casting, and duplicate detection. Once cleaned, the data should be written back to a new table in the same database and optionally exported to CSV so that downstream teams can verify the results in Excel if they choose.
Key deliverables
– Python script (Pandas, NumPy, SQLAlchemy preferred) that performs the full ETL/cleaning routine
– Re-usable functions or class-based structure so future data drops can be processed with one command
– Clear inline comments plus a short README explaining installation, execution, and configurable parameters
Acceptance criteria
1. Script connects to the database with credentials provided at run-time or via .env file
2. All missing or invalid numerical entries are handled according to the logic we agree on (e.g., imputation, removal)
3. Cleaned dataset is inserted into a new table without breaking existing schemas and passes a row-count parity check
4. Running time on the full dataset is under 10 minutes on a standard workstation
If you have experience building robust Python data-cleaning workflows and can demonstrate prior work with large numerical datasets, I’d like to get this moving quickly.
I expect the script to connect via SQL (SQLAlchemy or similar), ingest the relevant tables, and then handle missing values, outliers, type casting, and duplicate detection. Once cleaned, the data should be written back to a new table in the same database and optionally exported to CSV so that downstream teams can verify the results in Excel if they choose.
Key deliverables
– Python script (Pandas, NumPy, SQLAlchemy preferred) that performs the full ETL/cleaning routine
– Re-usable functions or class-based structure so future data drops can be processed with one command
– Clear inline comments plus a short README explaining installation, execution, and configurable parameters
Acceptance criteria
1. Script connects to the database with credentials provided at run-time or via .env file
2. All missing or invalid numerical entries are handled according to the logic we agree on (e.g., imputation, removal)
3. Cleaned dataset is inserted into a new table without breaking existing schemas and passes a row-count parity check
4. Running time on the full dataset is under 10 minutes on a standard workstation
If you have experience building robust Python data-cleaning workflows and can demonstrate prior work with large numerical datasets, I’d like to get this moving quickly.