Daily JSON Data Scraping
Budget: ₹600 – ₹1,500 INR
I run several ongoing web-data initiatives and need a reliable partner to handle the scraping side. Every day you will extract fresh data from a rotating set of target sites and deliver it as well-structured JSON.
The workflow should be fully automated (Python, Scrapy, BeautifulSoup, Selenium, or similar) and resilient to layout tweaks, CAPTCHAs, and IP blocks. I’ll provide the list of URLs and the specific fields required for each scrape; you’ll return parsed JSON plus a quick success log so I can drop the data straight into our pipelines.
Acceptance criteria
• Script or spider runs unattended on a schedule I can trigger via cron or similar
• Returned JSON matches the field names and hierarchy I specify for each site
• Error handling with retry logic and a summary report for any failed records
• Codebase, brief setup notes, and credentials handled securely in Git
This is a long-term arrangement, so clean, maintainable code and responsive communication are essential.
The workflow should be fully automated (Python, Scrapy, BeautifulSoup, Selenium, or similar) and resilient to layout tweaks, CAPTCHAs, and IP blocks. I’ll provide the list of URLs and the specific fields required for each scrape; you’ll return parsed JSON plus a quick success log so I can drop the data straight into our pipelines.
Acceptance criteria
• Script or spider runs unattended on a schedule I can trigger via cron or similar
• Returned JSON matches the field names and hierarchy I specify for each site
• Error handling with retry logic and a summary report for any failed records
• Codebase, brief setup notes, and credentials handled securely in Git
This is a long-term arrangement, so clean, maintainable code and responsive communication are essential.
Related categories:
JavaScript
Python
Web Scraping
Data Mining
JSON
Scrapy
Data Extraction
BeautifulSoup
Selenium
Automation