Daily d247 Data Scraper API
Budget: $30 – $250 USD
I need an automated pipeline that pulls fresh data from d247.com every 24 hours and serves it through a lightweight REST-style API in clean JSON. My ideal flow is:
• Headless scraper (Python + BeautifulSoup, Scrapy, or Playwright—open to your suggestion) that navigates the relevant pages, captures the required fields, and normalises them.
• A small backend (FastAPI or Node/Express are both fine) that stores each crawl in a local database and exposes endpoints for today’s data as well as a rolling archive.
• Simple scheduling—cron, GitHub Actions, or a cloud function—so the job runs automatically once a day and retries gracefully on failures.
Please include brief documentation covering how to deploy the scraper, run it locally, and hit the JSON endpoints. I’ll consider the work complete when I can trigger a crawl, watch it populate the database, and fetch the latest JSON response in under a second.
• Headless scraper (Python + BeautifulSoup, Scrapy, or Playwright—open to your suggestion) that navigates the relevant pages, captures the required fields, and normalises them.
• A small backend (FastAPI or Node/Express are both fine) that stores each crawl in a local database and exposes endpoints for today’s data as well as a rolling archive.
• Simple scheduling—cron, GitHub Actions, or a cloud function—so the job runs automatically once a day and retries gracefully on failures.
Please include brief documentation covering how to deploy the scraper, run it locally, and hit the JSON endpoints. I’ll consider the work complete when I can trigger a crawl, watch it populate the database, and fetch the latest JSON response in under a second.