Product Scraper for Custom Sites
Budget: ₹750 – ₹1,250 INR
I need complete product information extracted from several custom-built e-commerce sites. Titles, prices, SKU, stock status, image URLs, descriptions, and any variant data all have to be captured, cleansed, de-duplicated, and delivered to me in a single, well-formatted CSV.
Python is the preferred stack—BeautifulSoup for fast parsing, Selenium for the sections hidden behind dynamic elements or login gates, and Scrapy for the heavy lifting and crawl management. Feel free to combine or swap these libraries as long as the final dataset is accurate and the run time stays reasonable.
Timing is tight: I’d like the first pass within two days and the final validated file no later than day three.
Deliverables
• Clean CSV with every requested field populated and no empty rows
• Re-usable Python scripts (with comments) plus a short README describing how to run them and any environment variables or requirements.txt entries
• Quick validation summary showing total products found, duplicates removed, and any pages that failed to load after retries
Acceptance criteria
• ≥ 98 % field completion when spot-checked
• Pagination, lazy-loaded images, and JS-rendered prices successfully captured
• Scripts run on a fresh machine with only the documented dependencies installed
If that timeline and scope fit your current bandwidth, tell me your approach to anti-bot measures and how you plan to keep the crawl polite to each host (e.g., throttling, rotating headers, or proxy use).
Python is the preferred stack—BeautifulSoup for fast parsing, Selenium for the sections hidden behind dynamic elements or login gates, and Scrapy for the heavy lifting and crawl management. Feel free to combine or swap these libraries as long as the final dataset is accurate and the run time stays reasonable.
Timing is tight: I’d like the first pass within two days and the final validated file no later than day three.
Deliverables
• Clean CSV with every requested field populated and no empty rows
• Re-usable Python scripts (with comments) plus a short README describing how to run them and any environment variables or requirements.txt entries
• Quick validation summary showing total products found, duplicates removed, and any pages that failed to load after retries
Acceptance criteria
• ≥ 98 % field completion when spot-checked
• Pagination, lazy-loaded images, and JS-rendered prices successfully captured
• Scripts run on a fresh machine with only the documented dependencies installed
If that timeline and scope fit your current bandwidth, tell me your approach to anti-bot measures and how you plan to keep the crawl polite to each host (e.g., throttling, rotating headers, or proxy use).