Robust Web Scraping for Product Data Extraction

We are looking for an experienced web scraping developer to build a robust and scalable scraping system for extracting product data from G2A and integrating it into our Laravel-based platform.

We already have an integration with the G2A API. However, some critical data is NOT available or NOT accurate via the API, which is why scraping is required.

---

Project Scope:

* Scrape and process data for 240,000+ products
* Integrate scraping output with an existing Laravel backend
* Ensure data consistency and correct mapping with existing API data

---

Important Clarification (API vs Scraping):

* We already receive the following from G2A API:

* Product ID (same as G2A)
* Slug (same as G2A)
* Price, currency
* Images
* Platform, region
* System requirements

* However:

* Categories from the API are NOT accurate and do not match G2A website structure
* Some important content (description sections, alert boxes, activation guides) is NOT available via API

Therefore, scraping is required to:

* Rebuild the correct categories structure exactly as on G2A
* Extract missing product content

All scraped data must be correctly matched with API data using the same product ID and slug.

---

Core Requirements:

1. Categories Mapping (Very Important)

* Extract the full categories structure exactly as on G2A website
* Include main categories, subcategories, and full hierarchy
* Replace API categories completely
* Each product must be mapped accurately to its category path

2. Product Data Extraction

* Basic product information
* Full product description (HTML)
* Yellow alert / warning box under “About this item” (must be extracted بالكامل with formatting)
* Activation guide (especially for software products)

3. Data Alignment

* Must correctly match scraped data with API data (using product ID and slug)
* No duplicates or mismatched records

---

Technical Requirements:

* Python (preferred with FastAPI or similar)
* Experience with headless browsers (Playwright / Selenium / Puppeteer)
* Ability to handle dynamic content and anti-bot protections
* Experience with scalable scraping (parallel workers, batching, queues)
* Strong error handling, retry logic, and logging

---

Infrastructure:

* Scraper will run on a dedicated VPS (8 CPU / 32GB RAM)
* Must support parallel execution
* Must not affect the main Laravel application

---

Milestones:

1. Initial Test (Mandatory)

* Scrape 50 varied products (games, software, gift cards)
* Display results on the live website
* Validate:

* Data accuracy
* Categories mapping (must match G2A exactly)
* Alert box extraction
* Activation guide

2. Scaling Phase

* Gradual scaling after validation
* Full scraping of 240K+ products

---

Payment Terms:

* No upfront payment without results
* Payment after successful delivery of the initial test (50 products)
* Further payments based on validated milestones

---

To Apply:

Please include:

* Examples of similar scraping projects (especially large-scale)
* Technologies used
* Your approach for handling this project (short explanation)

---

We are looking for someone who can deliver real, scalable results — not partial implementations.