Website Data Scraper
Budget: $30 – $250 AUD
We need a robust yet lightweight script that can automatically pull business details from a publicly accessible government website. The information to capture will centre on business details, such as registration numbers, business name, registration date, address, etc.
The workflow should:
• Navigate every relevant section of the site (pagination, search filters, subsidiary pages).
• Extract the required fields accurately
• Export clean, structured data to CSV and JSON
A Python solution leveraging requests/BeautifulSoup or Scrapy is preferred, but I’m open to other dependable stacks if they handle rate-limits, retries, and potential CAPTCHA gracefully. The script must be easy to rerun on demand, with clear instructions for environment setup and any dependencies.
Acceptance criteria will be a sample scrape of 500 records that match the live site exactly, plus the commented source code and a short README outlining how to run, customise, and schedule the job.
The workflow should:
• Navigate every relevant section of the site (pagination, search filters, subsidiary pages).
• Extract the required fields accurately
• Export clean, structured data to CSV and JSON
A Python solution leveraging requests/BeautifulSoup or Scrapy is preferred, but I’m open to other dependable stacks if they handle rate-limits, retries, and potential CAPTCHA gracefully. The script must be easy to rerun on demand, with clear instructions for environment setup and any dependencies.
Acceptance criteria will be a sample scrape of 500 records that match the live site exactly, plus the commented source code and a short README outlining how to run, customise, and schedule the job.
Related categories:
JavaScript
Python
Data Processing
Web Scraping
Software Architecture
JSON
Scrapy
Data Extraction
BeautifulSoup
Automation