Hiring Developer: Universal Product Page Scraper (JS-rendered sites) + GPT-assisted extraction + structured JSON output -- 2

Hey! I’m looking to hire an experienced developer to build a universal product-detail scraping pipeline that takes a product URL (any website) and returns a complete structured product record.

This is not a “simple HTML parse.” Many target sites are React/Next/Vue, load content via XHR/GraphQL, hide details behind tabs/accordions/modals, and lazy-load images/PDFs. The solution needs to reliably extract everything a human can see on the page, plus the underlying data used to render it.

What the scraper must do (high level)

Given a product URL, the pipeline should:

Load the page like a real user (handle cookies/overlays).

Capture all content from multiple sources (DOM + network + interactions).

Use GPT API strategically to increase accuracy (field mapping, variant extraction, doc classification, completeness checks).

Output a strict, validated JSON record + optional Excel export.

Data fields I need to extract (core)

Required output fields:

Product name

manufacturer / brand

description (clean, human-readable)

images[] (high quality URLs, deduped; include context/alt when possible)

documents[] (PDF/spec sheets/install guides/warranties/BIM/etc., classified)

options[] / variants (SKUs if available; option dimensions like color/size/material; availability if available)

attributes{} (everything else: specs, dimensions, sustainability/certifications, compliance info, finish codes, etc.)

Additionally (for completeness & auditability):

Full rendered page text: pageText

Sectioned text: headings/paragraphs/lists/tables: pageTextSections

Structured data capture: JSON-LD + embedded state blobs (e.g., __NEXT_DATA__) + meta tags

Network payload evidence: selected API responses that contain product truth (saved with URL + snippet/hash)

Provenance per field: source + confidence + evidence snippet

Universal extraction approach I want (technical requirements)

Tech stack (preferred):

Playwright (preferred) or Puppeteer for browser automation

Node.js or Python acceptable

GPT API integration for: mapping, variants, document classification, and completeness audit loops

Must-have capabilities:

JS-rendered content support (wait for hydration; not just raw HTML)

Network interception:

capture JSON/GraphQL responses during load + interactions

Interaction replay:

scroll for lazy loads

expand accordions (“See more”, “Specs”, “Downloads”)

click tabs

open modals/drawers (e.g., availability, downloads)

attempt variant selection and record deltas

Asset harvesting:

harvest images & PDFs from DOM and network responses (not only <a href> / <img src>)

Anti-fragility:

robust waiting (not only networkidle)

retry logic

consistent error reporting

Output validation:

JSON schema validation

deterministic structure even when fields are missing (nulls, empty arrays)

How GPT should be used (important)

I have a GPT API key and want it used heavily but intelligently:

Decide page type and extraction plan (product vs category vs doc page)

Identify which network payloads contain product data

Normalize messy specs into key/value

Reconstruct variants/options from partial signals

Classify documents (spec sheet vs install vs warranty vs BIM)

Run a completeness audit and suggest the next actions (click this / expand that) until the record is complete

Rule: GPT must not hallucinate. If uncertain, output null + evidence + recommended next action.

Deliverables

A runnable scraper (CLI or small service) that accepts a product URL and outputs:

product.json (structured)

optional product.xlsx

A “self-healing” completeness loop with logs:

what interactions were performed

what was missing

what sources were used (DOM/network/GPT/OCR if used)

Documentation:

setup instructions

how to add sites / tune extraction

how to run in headless mode

Basic test set:

run on ~10 diverse product URLs (Shopify + custom + Next.js + heavy JS) and show outputs

Nice-to-haves

Dockerfile

Queue/scheduler support for batch runs

Proxy support (only if needed)

Optional OCR fallback using screenshot + vision for hard edge cases