Nationwide Assistance Sites Data Scraping
Budget: $250 – $750 USD
I need a diligent data-scraper to collect structured information from both public assistance and emergency remediation websites for all 50 states, the District of Columbia, and Puerto Rico—52 jurisdictions in total. For every jurisdiction you will pull the same 43 data points; templates that show the exact field names, formatting rules, and example values are ready for you, along with a starter list of recommended source URLs.
Scope and priorities
• Treat public assistance and emergency remediation sources with equal importance, making sure no jurisdiction is missing either category.
• Accuracy around service details is paramount. Those fields (program name, description, scope, cost coverage, languages served, etc.) must be exact; other fields such as contact information and eligibility still need to be captured but may be filled after you have locked in the core service content.
• Scraping must observe each site’s robots.txt and rate-limit guidance. Where a site blocks automated collection, fallback to respectful manual extraction is fine.
Workflow
1. Locate the authoritative state or territorial pages for both service categories.
2. Extract the 43 required data points, normalising any odd formats (e.g., multi-line addresses, embedded phone numbers).
3. Validate and store each jurisdiction’s data in its own JSON file, following the supplied naming convention (e.g., “CA_public_services.json”).
4. Run JSON-lint or an equivalent validator before delivery so that every file loads cleanly into the pSEO import tool.
Preferred stack
Python with BeautifulSoup, Scrapy, or Selenium is ideal, but you’re free to use any tooling that reliably outputs clean JSON. An existing script library can be shared if that accelerates your process.
Deliverables
• 52 individual, schema-compliant JSON files (one per jurisdiction).
• A brief changelog or scraping log noting the final URL sources for each file and any anomalies you encountered.
Acceptance will be based on:
– Presence of all 43 data fields in every JSON file.
– Accuracy and completeness of service-detail sections.
– Zero schema or validation errors.
If you have run large-scale scrapes before and can turn this around quickly without sacrificing quality, I’d love to see your approach.
Scope and priorities
• Treat public assistance and emergency remediation sources with equal importance, making sure no jurisdiction is missing either category.
• Accuracy around service details is paramount. Those fields (program name, description, scope, cost coverage, languages served, etc.) must be exact; other fields such as contact information and eligibility still need to be captured but may be filled after you have locked in the core service content.
• Scraping must observe each site’s robots.txt and rate-limit guidance. Where a site blocks automated collection, fallback to respectful manual extraction is fine.
Workflow
1. Locate the authoritative state or territorial pages for both service categories.
2. Extract the 43 required data points, normalising any odd formats (e.g., multi-line addresses, embedded phone numbers).
3. Validate and store each jurisdiction’s data in its own JSON file, following the supplied naming convention (e.g., “CA_public_services.json”).
4. Run JSON-lint or an equivalent validator before delivery so that every file loads cleanly into the pSEO import tool.
Preferred stack
Python with BeautifulSoup, Scrapy, or Selenium is ideal, but you’re free to use any tooling that reliably outputs clean JSON. An existing script library can be shared if that accelerates your process.
Deliverables
• 52 individual, schema-compliant JSON files (one per jurisdiction).
• A brief changelog or scraping log noting the final URL sources for each file and any anomalies you encountered.
Acceptance will be based on:
– Presence of all 43 data fields in every JSON file.
– Accuracy and completeness of service-detail sections.
– Zero schema or validation errors.
If you have run large-scale scrapes before and can turn this around quickly without sacrificing quality, I’d love to see your approach.
Related categories:
Data Entry
Excel
Web Scraping
Data Mining
JSON
Scrapy
Data Scraping
BeautifulSoup
Selenium