E-commerce Data Scraper & MySQL Integrator
Budget: £250 – £750 GBP
Robust Web Scraper + MySQL Integration for Product Data Collection
** Automatic responses will not be considered **
We require a robust, maintainable, and efficient web scraping solution that can extract product URLs and associated product data from multiple retail websites, then store the results directly into a MySQL database.
Key Requirements:
Multi-site Scraping Capability
Ability to handle multiple retail websites (we will provide the initial target sites).
Configuration-driven scraping — adding a new site should only require entering selectors, not editing core code.
Must handle pagination, category navigation, and product variants (sizes, colours, etc.).
Our team must be able to specify the CSS/XPath selectors for product pages via a configuration file or database entry.
Once selectors are set for a site, the scraper should extract the required fields from every product page on that site without additional coding.
Data to Collect:
Product URL
SKU / Product Code
EAN / GTIN
Title / Product Name
Price (numeric value only)
Any promotional price or discount info
Stock availability
Image URLs
Product attributes (size, colour, etc.)
MySQL Database Integration:
Directly insert/update into an existing MySQL 8.0 database.
Avoid duplicate entries (check by SKU or URL).
Performance & Reliability
Must handle large product catalogs without timeouts or excessive load on target sites.
Respect robots.txt where required, but be able to use proxies/rotating IPs if needed for public data.
Retry logic for failed requests.
Instructions for deployment on Windows Server with WAMP (PHP) or Python + MySQL. Other technologies will be considered
Deliverables
Fully working scraper with code & dependencies.
MySQL schema creation/alteration scripts.
Step-by-step deployment instructions.
Brief documentation on adding new sites and updating selectors.
Our budget is a placeholder. Applicants will be expected to adhere to their bid price unless requirements change, so ask as many questions as needed before we make our decision, not after.
** Automatic responses will not be considered **
We require a robust, maintainable, and efficient web scraping solution that can extract product URLs and associated product data from multiple retail websites, then store the results directly into a MySQL database.
Key Requirements:
Multi-site Scraping Capability
Ability to handle multiple retail websites (we will provide the initial target sites).
Configuration-driven scraping — adding a new site should only require entering selectors, not editing core code.
Must handle pagination, category navigation, and product variants (sizes, colours, etc.).
Our team must be able to specify the CSS/XPath selectors for product pages via a configuration file or database entry.
Once selectors are set for a site, the scraper should extract the required fields from every product page on that site without additional coding.
Data to Collect:
Product URL
SKU / Product Code
EAN / GTIN
Title / Product Name
Price (numeric value only)
Any promotional price or discount info
Stock availability
Image URLs
Product attributes (size, colour, etc.)
MySQL Database Integration:
Directly insert/update into an existing MySQL 8.0 database.
Avoid duplicate entries (check by SKU or URL).
Performance & Reliability
Must handle large product catalogs without timeouts or excessive load on target sites.
Respect robots.txt where required, but be able to use proxies/rotating IPs if needed for public data.
Retry logic for failed requests.
Instructions for deployment on Windows Server with WAMP (PHP) or Python + MySQL. Other technologies will be considered
Deliverables
Fully working scraper with code & dependencies.
MySQL schema creation/alteration scripts.
Step-by-step deployment instructions.
Brief documentation on adding new sites and updating selectors.
Our budget is a placeholder. Applicants will be expected to adhere to their bid price unless requirements change, so ask as many questions as needed before we make our decision, not after.
Related categories:
PHP
Python
Web Scraping
Software Architecture
MySQL
Scrapy
BeautifulSoup
Data Integration
Automation
Data Collection