Robust PySpark ETL & SQL Analytics Pipeline
Budget: ₹1,500 – ₹12,500 INR
Project Title: Build End-to-End Data Cleaning, ETL Pipeline & SQL Analytics (PySpark)
I need a skilled Data Engineer / Data Analyst to build a complete end-to-end data pipeline using the raw CSV files provided. The project involves data cleaning, transformation, building a star schema, implementing ETL logic in PySpark, writing analytical SQL queries, and performing data quality checks.
The files included are:
users.csv (dirty user data – nulls, duplicates, inconsistent casing)
products.csv (messy categories and SKU formatting)
orders.csv (20k+ orders with mixed date formats, invalid numeric fields)
order_items.csv (dirty SKUs, wrong quantities, duplicates)
Scope of Work:
Data Cleaning & Standardization
Fix inconsistent casing, extra spaces, special characters
Convert fields into proper numeric & datetime formats
Handle NULL values and duplicates
Validate & correct foreign keys (user_id, product_id, order_id, sku)
Data Modeling – Star Schema
Build fact_orders
Build dim_users, dim_products, dim_date
Ensure all joins and relationships are valid
ETL Pipeline (PySpark)
Load raw CSVs
Apply cleaning and transformation logic
Generate clean, analysis-ready datasets
Save cleaned output CSVs
SQL Analytical Queries (Mandatory)
Examples:
Orders placed from selected cities and date ranges
Monthly revenue trend
Highest selling product/category
Repeat customers
Average order value per user
City-wise sales contribution
Data Quality Checks
Null checks
Duplicate checks
Referential integrity validation
Data type validation
Final Deliverables:
Cleaned users.csv
Cleaned products.csv
Cleaned orders.csv
Cleaned order_items.csv
Completed PySpark/Jupyter Notebook
Star Schema ERD
SQL queries + outputs
Data Quality Summary
Required Skills:
PySpark / Python
SQL
ETL pipeline development
Data cleaning & wrangling
Data modeling (Star Schema)
Jupyter Notebook / Google Colab experience
What to Include in Your Proposal:
Relevant experience with PySpark or Data Engineering
Similar projects you have done
Estimated delivery time & cost
I need a skilled Data Engineer / Data Analyst to build a complete end-to-end data pipeline using the raw CSV files provided. The project involves data cleaning, transformation, building a star schema, implementing ETL logic in PySpark, writing analytical SQL queries, and performing data quality checks.
The files included are:
users.csv (dirty user data – nulls, duplicates, inconsistent casing)
products.csv (messy categories and SKU formatting)
orders.csv (20k+ orders with mixed date formats, invalid numeric fields)
order_items.csv (dirty SKUs, wrong quantities, duplicates)
Scope of Work:
Data Cleaning & Standardization
Fix inconsistent casing, extra spaces, special characters
Convert fields into proper numeric & datetime formats
Handle NULL values and duplicates
Validate & correct foreign keys (user_id, product_id, order_id, sku)
Data Modeling – Star Schema
Build fact_orders
Build dim_users, dim_products, dim_date
Ensure all joins and relationships are valid
ETL Pipeline (PySpark)
Load raw CSVs
Apply cleaning and transformation logic
Generate clean, analysis-ready datasets
Save cleaned output CSVs
SQL Analytical Queries (Mandatory)
Examples:
Orders placed from selected cities and date ranges
Monthly revenue trend
Highest selling product/category
Repeat customers
Average order value per user
City-wise sales contribution
Data Quality Checks
Null checks
Duplicate checks
Referential integrity validation
Data type validation
Final Deliverables:
Cleaned users.csv
Cleaned products.csv
Cleaned orders.csv
Cleaned order_items.csv
Completed PySpark/Jupyter Notebook
Star Schema ERD
SQL queries + outputs
Data Quality Summary
Required Skills:
PySpark / Python
SQL
ETL pipeline development
Data cleaning & wrangling
Data modeling (Star Schema)
Jupyter Notebook / Google Colab experience
What to Include in Your Proposal:
Relevant experience with PySpark or Data Engineering
Similar projects you have done
Estimated delivery time & cost