SEO Data Scraping Bot Development
Budget: $250 – $750 USD
I’m looking for an experienced developer to build a scraping / crawling bot that will run on my own server (Linux / cloud VM). This bot will continuously gather SEO & competitor data (local + non-local) from many sources (websites, directories, business listings, etc.) and store it in a structured database.
Key deliverables / tasks:
Design and implement the scraper/crawler architecture (e.g. using Python, Scrapy, Selenium, etc.).
Support dynamic pages, AJAX-loaded content, pagination, and elements behind logins if needed (if legitimate).
Use rotating proxies, rate limiting, user agents, and anti-block techniques to avoid bans.
Store results into a database (you choose SQL / NoSQL) with schema optimized for local vs non-local data.
Provide an API (or query interface) so my AI SEO tool can fetch data (e.g. via REST or internal functions).
Include logging, error recovery, retry logic, and scheduler (cron or task queue).
Provide documentation on how to deploy, maintain, add new sources, and troubleshoot.
(Optional but preferred) Build an admin dashboard or monitoring interface to see scraping stats, failures, data health.
Skills & experience required:
Strong Python (or Node.js) skills, particularly with web scraping frameworks (Scrapy, BeautifulSoup, Selenium, Puppeteer, etc.).
Experience handling anti-scraping defenses, captchas, IP rotation, etc.
Database design (SQL / NoSQL) and ETL / data pipelines.
API development experience.
Familiarity with deploying on servers / cloud (AWS, DigitalOcean, VPS, etc.).
Clear documentation, version control (Git), and good communication skills.
Key deliverables / tasks:
Design and implement the scraper/crawler architecture (e.g. using Python, Scrapy, Selenium, etc.).
Support dynamic pages, AJAX-loaded content, pagination, and elements behind logins if needed (if legitimate).
Use rotating proxies, rate limiting, user agents, and anti-block techniques to avoid bans.
Store results into a database (you choose SQL / NoSQL) with schema optimized for local vs non-local data.
Provide an API (or query interface) so my AI SEO tool can fetch data (e.g. via REST or internal functions).
Include logging, error recovery, retry logic, and scheduler (cron or task queue).
Provide documentation on how to deploy, maintain, add new sources, and troubleshoot.
(Optional but preferred) Build an admin dashboard or monitoring interface to see scraping stats, failures, data health.
Skills & experience required:
Strong Python (or Node.js) skills, particularly with web scraping frameworks (Scrapy, BeautifulSoup, Selenium, Puppeteer, etc.).
Experience handling anti-scraping defenses, captchas, IP rotation, etc.
Database design (SQL / NoSQL) and ETL / data pipelines.
API development experience.
Familiarity with deploying on servers / cloud (AWS, DigitalOcean, VPS, etc.).
Clear documentation, version control (Git), and good communication skills.