E-commerce Website Data Scraping in Python
Budget: $10 – $30 USD
We have a backlog of online retailers to scrape. This is the first job, and if the outcome, timeline, communications and ways of work are satisfactory, there will be more jobs to come in the long term.
The freelancer will create a Python website scrape script using Beautiful Soup 4 (preferred) or scrapy (less preferred) or Selenium (only if Beautiful Soup and scrapy cannot complete the scrape job). The scrape script should create 2 CSV files:
File 1: product_list_table. This includes all product names, product_url, retailer name. Each product should only appear once in this file
File 2: product_details_table. This scrapes the product details page of each product. If a product has a numver of variants or product sizes (e.g. lipsticks - red, purple, green; energy drink - 250ml, 330ml, 500ml), then there should be 1 entry for each variant.
For each retailer, we may request that only specific categories to be scraped, or all the brands to be scraped.
We have included sample output CSV files - please see attached.
The output script should be:
- Clean
- Efficient - we expect concurrency to be used to reduce scrape time
- Complete - all products and product details are recorded correctly and with accuracy
- Prepares for edge cases - it should be designed to support scenarios where there are a few HTML formats within a retailer
- Supports anti-scrape - we have proxy solutions. They should be utilised where necessary
- Supports de-bugging - showing progress as the script runs (e.g. x products out of y left), collecting products which are not scraped, having log files for de-bugging
- Prints out progress and error messages or creates a log file so that debugging and failed scenarios could be observed and fixed
The freelancer will create a Python website scrape script using Beautiful Soup 4 (preferred) or scrapy (less preferred) or Selenium (only if Beautiful Soup and scrapy cannot complete the scrape job). The scrape script should create 2 CSV files:
File 1: product_list_table. This includes all product names, product_url, retailer name. Each product should only appear once in this file
File 2: product_details_table. This scrapes the product details page of each product. If a product has a numver of variants or product sizes (e.g. lipsticks - red, purple, green; energy drink - 250ml, 330ml, 500ml), then there should be 1 entry for each variant.
For each retailer, we may request that only specific categories to be scraped, or all the brands to be scraped.
We have included sample output CSV files - please see attached.
The output script should be:
- Clean
- Efficient - we expect concurrency to be used to reduce scrape time
- Complete - all products and product details are recorded correctly and with accuracy
- Prepares for edge cases - it should be designed to support scenarios where there are a few HTML formats within a retailer
- Supports anti-scrape - we have proxy solutions. They should be utilised where necessary
- Supports de-bugging - showing progress as the script runs (e.g. x products out of y left), collecting products which are not scraped, having log files for de-bugging
- Prints out progress and error messages or creates a log file so that debugging and failed scenarios could be observed and fixed