Industrial Automation Data Extraction & Image Collection

Industrial Automation Product Data Extraction, Deduplication & Structured Image Collection

Project Overview
We are an industrial automation parts distributor building a structured product database to support inbound enquiries and SEO growth.

We require an experienced data extraction specialist to:

Extract structured product data from major industrial / electronic component distributor websites

Identify duplicate manufacturer part numbers across multiple sources

Merge all unique information into a single consolidated dataset

Extract and organise all available product images per part number

Deliver a clean, deduplicated, production-ready dataset

This project includes:

Data extraction

Normalization

Deduplication

Intelligent merging

Structured image collection and organisation

Accuracy and clean structure are critical.

Core Requirement – Deduplication & Data Merging
Many distributor websites list the same Manufacturer Part Number (MPN) but with different technical attributes.

We require:

One single row per unique Manufacturer Part Number

No duplicate MPNs in the final dataset

All unique specifications merged into that single row

Missing data from one source supplemented from another

If two sources provide different values for the same spec, both values must be preserved

No overwriting of valid data

This is not a simple scrape-and-export task. It requires structured consolidation.

Required Fields (Where Available)
For each unique MPN:

Manufacturer Part Number (normalized format)

Brand / Manufacturer

Product Description

Product Category

Product Status (Active / Obsolete / Discontinued)

Technical Specifications (structured)

Electrical specs (voltage, current, range, etc.)

Mechanical specs (if applicable)

Input / Output type

Datasheet link(s)

Source URLs (all sources used for that part)

Specifications must be structured — no raw HTML dumps.

Image Collection Requirements (Strict)
For each unique Manufacturer Part Number:

Extract all available product images from all sources

No watermarks

Highest available resolution

No thumbnails unless full resolution is unavailable

Folder Structure
Each part number must have its own folder:

Example:

Folder name:
1746-OB16

Inside folder:
1746-OB16 A.jpg
1746-OB16 B.jpg
1746-OB16 C.jpg
1746-OB16 D.jpg
1746-OB16 E.jpg
1746-OB16 F.jpg
1746-OB16 G.jpg

If 3 images exist on one source and 4 on another, all 7 must be included.

Naming convention:
MPN + space + alphabetical suffix

No duplicate image files.
No random filenames.
No inconsistent naming.

Output Requirements
1. Master Excel File (.xlsx)
One row per unique MPN

All specifications merged

Clean structured columns

No duplicate MPNs

No HTML tags

No formatting issues

2. Image Directory
Root folder containing subfolders for each MPN

Each subfolder named exactly as the MPN

All images inside correctly named (A, B, C…)

Clean file structure

Data Handling Expectations
The freelancer must:

Normalize part numbers (remove spacing / hyphen inconsistencies)

Identify duplicates across multiple sources

Merge specifications intelligently

Preserve all unique attributes

Handle dynamic / JS-rendered sites

Deliver clean, production-ready output

In your proposal, explain:

Your deduplication logic

Your merging methodology

How you will handle conflicting specifications

How you will prevent image duplication

Tools you will use

Estimated scale

Timeline

Total cost

Experience Required
Only apply if you have:

Experience scraping large distributor / e-commerce platforms

Experience handling structured product datasets

Experience deduplicating and merging datasets

Experience bulk-downloading and structuring images

Ability to deliver clean, organised, ready-to-import data

Please include examples of similar projects.

Project Goal
We are building a large structured industrial component database. If this phase is successful, this will become an ongoing project.