Expert Python Data Scraper
Budget: $30 – $250 USD
I need a seasoned Python developer to build a robust scraper that collects the required data and writes it straight to JSON—no additional cleaning or processing necessary. Once we begin I’ll provide the target URL(s) and any access details; for now, assume a standard public site with pagination and occasional anti-bot checks.
Core expectations
• Written in Python 3 using requests/BeautifulSoup or Scrapy; resort to Selenium only if there’s no lighter workaround.
• Handles pagination, retries, and polite delays gracefully so the run can complete unattended.
• Config file or clear constants for headers, cookies, and start URLs, letting me tweak targets without editing core logic.
• Produces a single JSON file (or one file per page if that’s cleaner) reflecting the raw content exactly as captured.
Deliverables
1. Fully commented source code.
2. README with quick-start setup and sample command.
3. Example JSON output generated by the script.
If you’ve tackled heavy-duty scraping before and can turn this around quickly, I’d love to hear your approach and estimated timeline.
Core expectations
• Written in Python 3 using requests/BeautifulSoup or Scrapy; resort to Selenium only if there’s no lighter workaround.
• Handles pagination, retries, and polite delays gracefully so the run can complete unattended.
• Config file or clear constants for headers, cookies, and start URLs, letting me tweak targets without editing core logic.
• Produces a single JSON file (or one file per page if that’s cleaner) reflecting the raw content exactly as captured.
Deliverables
1. Fully commented source code.
2. README with quick-start setup and sample command.
3. Example JSON output generated by the script.
If you’ve tackled heavy-duty scraping before and can turn this around quickly, I’d love to hear your approach and estimated timeline.
Related categories:
PHP
Python
Web Scraping
Software Architecture
JSON
Scrapy
Data Scraping
BeautifulSoup