Website API & Scraping Support

My website relies on fresh third-party data that is not readily delivered in a single feed, so I need a unified solution that both connects to external APIs and scrapes selected pages when an endpoint is unavailable. The goal is a seamless pipeline that collects, normalises and stores the data so my own code can display it in real time.

Here is the workflow I have in mind:

• API layer – connect to any available REST or GraphQL endpoints, handle authentication keys or tokens, manage rate limits, and return clean JSON.
• Scraper fallback – when no API exists, run a headless scraper (Python + Scrapy/BeautifulSoup or comparable stack) with proxy rotation and basic anti-bot evasion, then map the results to the same data structure used by the API calls.
• Data handling – write to my existing MySQL database, flag updates, and trigger my site’s content refresh.
• Automation & monitoring – schedule jobs via cron or a lightweight queue, log successes/failures, and send alerts if anything breaks.
• Documentation – concise setup notes and inline comments so I can maintain or extend the scripts later.

A short Zoom/Meet walkthrough at hand-over will finish the engagement. If you’ve built similar hybrid API/scraping solutions and can demonstrate reliably structured output, I’m ready to get started straight away.