Multi-Source Product Catalog Data Harmonization
Budget: ₹600 – ₹1,500 INR
Background:
A global e-commerce platform aggregates product listings from thousands of vendors worldwide. Each vendor submits product data in CSV format, but the structure, formatting, and quality vary drastically. For example:
Some vendors use `price_usd`, others use `price` (with no currency), or `price (EUR)`.
Product categories are inconsistent: `Electronics > Phones`, `Tech/Mobiles`, or `Gadgets`.
Missing fields (e.g., `weight_kg`), duplicate entries, or free-text fields with typos (e.g., `color: reD`, `colour: crimson`).
Your task is to design a data harmonization engine that transforms these heterogeneous CSV files into a standardized schema while preserving accuracy and usability.
Objectives
1. Schema Mapping: Map varying column names (e.g., `price_usd`, `price (EUR)`) to a standardized format (`price`, `currency`).
2. Data Cleaning: Handle missing fields, typos, duplicates, and inconsistent units (e.g., `weight_kg` vs. `weight_lb`).
3. Category Normalization: Resolve ambiguous categories (e.g., `Tech/Mobiles` → `Electronics/Phones`) using a mix of rules and fuzzy logic.
4. Scalability: Ensure the solution can process 10,000+ rows efficiently.
Input/Output
- Input: A CSV file with arbitrary column names and mixed-quality data.
- Output: A cleaned CSV adhering to the [standard schema](link_to_schema_documentation) and a log file detailing transformations/errors.
Constraints
- You cannot assume prior knowledge of vendor-specific column names (e.g., a new vendor might use `cost` instead of `price`).
- The solution must be explainable (no black-box AI models).
Evaluation Criteria
1. Accuracy: How well does the output align with the standardized schema?
2. Flexibility: Can the engine adapt to new vendors without code changes?
3. Performance: Efficiency in handling large datasets.
4. Creativity: Unique approaches to mapping, error handling, or category normalization.
Deliverables
1. A working script or tool.
2. A brief document explaining your design choices, trade-offs, and how to extend the solution.
3. (Optional) A visualization of the data flow or error rates across test cases.
A global e-commerce platform aggregates product listings from thousands of vendors worldwide. Each vendor submits product data in CSV format, but the structure, formatting, and quality vary drastically. For example:
Some vendors use `price_usd`, others use `price` (with no currency), or `price (EUR)`.
Product categories are inconsistent: `Electronics > Phones`, `Tech/Mobiles`, or `Gadgets`.
Missing fields (e.g., `weight_kg`), duplicate entries, or free-text fields with typos (e.g., `color: reD`, `colour: crimson`).
Your task is to design a data harmonization engine that transforms these heterogeneous CSV files into a standardized schema while preserving accuracy and usability.
Objectives
1. Schema Mapping: Map varying column names (e.g., `price_usd`, `price (EUR)`) to a standardized format (`price`, `currency`).
2. Data Cleaning: Handle missing fields, typos, duplicates, and inconsistent units (e.g., `weight_kg` vs. `weight_lb`).
3. Category Normalization: Resolve ambiguous categories (e.g., `Tech/Mobiles` → `Electronics/Phones`) using a mix of rules and fuzzy logic.
4. Scalability: Ensure the solution can process 10,000+ rows efficiently.
Input/Output
- Input: A CSV file with arbitrary column names and mixed-quality data.
- Output: A cleaned CSV adhering to the [standard schema](link_to_schema_documentation) and a log file detailing transformations/errors.
Constraints
- You cannot assume prior knowledge of vendor-specific column names (e.g., a new vendor might use `cost` instead of `price`).
- The solution must be explainable (no black-box AI models).
Evaluation Criteria
1. Accuracy: How well does the output align with the standardized schema?
2. Flexibility: Can the engine adapt to new vendors without code changes?
3. Performance: Efficiency in handling large datasets.
4. Creativity: Unique approaches to mapping, error handling, or category normalization.
Deliverables
1. A working script or tool.
2. A brief document explaining your design choices, trade-offs, and how to extend the solution.
3. (Optional) A visualization of the data flow or error rates across test cases.