Scalable Indeed Job Scraper
Budget: $30 – $250 USD
I need a production-ready scraper that keeps my Indeed dataset fresh every hour. The crawler should pull every field I have identified—job title, company name, official company website, full description, salary, skills, experience, location, apply URL and posted date—while de-duplicating anything already stored. Of those, job title and company name must never be missed because they anchor the rest of my pipeline.
I would like the core built in Python and I am comfortable if you reach for Playwright, Puppeteer, Scrapy or Selenium so long as the codebase stays clean and readable. Results should flow straight into either PostgreSQL or MongoDB; future scale is important, so please structure tables/collections with growth in mind.
Indeed can be unforgiving, so the scraper has to rotate proxies by default. Feel free to layer on further anti-bot tactics (stealth headless settings, adaptive delays, etc.), but proxy rotation is the non-negotiable starting point.
When a listing does not mention the company’s website, the scraper should still track it down—whether through an extra search pass or by parsing the apply URL—so every record ends with an accurate domain.
Deliverables
• Fully functional scraper with hourly auto-refresh
• Duplicate-detection logic and company-domain lookup
• Database integration script (PostgreSQL or MongoDB)
• Configuration for proxy rotation and other anti-blocking measures
• Clear, step-by-step deployment guide with environment requirements
In your proposal, show me similar scraping projects you have tackled, outline your preferred tech stack, describe the anti-blocking flow you plan to implement, and give me an honest timeline from first commit to live deployment.
I would like the core built in Python and I am comfortable if you reach for Playwright, Puppeteer, Scrapy or Selenium so long as the codebase stays clean and readable. Results should flow straight into either PostgreSQL or MongoDB; future scale is important, so please structure tables/collections with growth in mind.
Indeed can be unforgiving, so the scraper has to rotate proxies by default. Feel free to layer on further anti-bot tactics (stealth headless settings, adaptive delays, etc.), but proxy rotation is the non-negotiable starting point.
When a listing does not mention the company’s website, the scraper should still track it down—whether through an extra search pass or by parsing the apply URL—so every record ends with an accurate domain.
Deliverables
• Fully functional scraper with hourly auto-refresh
• Duplicate-detection logic and company-domain lookup
• Database integration script (PostgreSQL or MongoDB)
• Configuration for proxy rotation and other anti-blocking measures
• Clear, step-by-step deployment guide with environment requirements
In your proposal, show me similar scraping projects you have tackled, outline your preferred tech stack, describe the anti-blocking flow you plan to implement, and give me an honest timeline from first commit to live deployment.