E-Commerce Image Scraping Automation
Budget: €30 – €250 EUR
I need a robust Python script that automatically collects product images from a range of e-commerce sites. The workflow should rely on Selenium where dynamic content or lazy-loading occurs, and fall back to lightweight Requests + BeautifulSoup when a simple GET will do.
Key expectations
• The script must crawl category and product pages I specify, detect all image URLs (including those generated by JavaScript), download the files, and save them locally with meaningful names.
• It should respect robots.txt rules and allow me to throttle request speed to avoid blocks.
• Configuration (target URLs, output folder, delay times, user-agent) should sit in a single JSON or YAML file so I can tweak it without touching the code.
• Output a concise log/report—how many pages visited, images found, duplicates skipped, and any failures.
Acceptance criteria
– Running `python main.py config.yml` successfully downloads every visible product image from the provided sample site.
– No 404 or corrupted files in the output directory.
– Code is clearly commented and passes a basic flake8 check.
Please keep the solution pure Python (no paid APIs) and compatible with Python 3.10.
Key expectations
• The script must crawl category and product pages I specify, detect all image URLs (including those generated by JavaScript), download the files, and save them locally with meaningful names.
• It should respect robots.txt rules and allow me to throttle request speed to avoid blocks.
• Configuration (target URLs, output folder, delay times, user-agent) should sit in a single JSON or YAML file so I can tweak it without touching the code.
• Output a concise log/report—how many pages visited, images found, duplicates skipped, and any failures.
Acceptance criteria
– Running `python main.py config.yml` successfully downloads every visible product image from the provided sample site.
– No 404 or corrupted files in the output directory.
– Code is clearly commented and passes a basic flake8 check.
Please keep the solution pure Python (no paid APIs) and compatible with Python 3.10.