Long-Term Web Scraping Engineer Needed
Budget: $3,000 – $5,000 USD
Web Scraping Engineer (Scrapy + Zyte API) – Long-Term Project
Overview
We are building a large-scale data aggregation system across 200+ listing / marketplace websites and are looking for an experienced web scraping engineer to own and maintain the scraping layer.
This is a serious, long-term project focused on reliability, cost control, and clean data.
Responsibilities
Build and maintain Scrapy spiders (one per site or site family)
Integrate Zyte API / Zyte Smart Proxy Manager
Extract structured data:
Address / location
Price (hourly / daily / monthly where applicable)
Listing URL
Handle:
Pagination / infinite scroll
JS-heavy sites (React / Vue / Angular)
XHR / JSON endpoint discovery
Export normalized JSON
Monitor and fix spiders when websites change
Optimize Zyte usage to minimize costs
Technical Requirements (Must-Have)
Strong experience with Scrapy
Experience using Zyte API or similar proxy/rendering services
Proven ability to scrape JavaScript-rendered websites
Clean, maintainable Python code
Understanding of robots.txt and ethical scraping
Git-based workflow
No Selenium-only approaches
No “browser rendering for everything”
Nice to Have
Marketplace / booking / real-estate scraping experience
XHR / API reverse-engineering skills
Cost-optimization mindset
Playwright knowledge (fallback only)
Deliverables
Scrapy spiders (clean, modular)
JSON exports per website
Configuration for:
Entry URLs
Pagination logic
Zyte usage
Basic documentation for maintenance
Project Scale
~200 websites total
Some sites: hundreds of pages
Others: tens of thousands of listings
Ongoing maintenance and expansion expected
Scrapy framework
Zyte integration
Standard JSON schema
Monthly retainer available for maintenance
We are cost-conscious and expect efficient Zyte usage.
Required Screening Questions (Do Not Skip)
In one sentence, explain the difference between Scrapy HTTP requests and browser rendering.
Name one JS-heavy website you have scraped and explain briefly how you extracted the data.
List two techniques to reduce Zyte API costs.
(Optional but recommended) Paste a short Scrapy spider skeleton (10–20 lines).
Generic answers will be rejected.
How to Apply
Please include:
Years of Scrapy experience
Example projects or GitHub (if available)
Your preferred pricing model (per-site / hourly / retainer)
Your estimated time per website
Why Work With Us?
Long-term collaboration
Clear technical direction
No shady scraping
Real production-scale system
Room to grow into a lead role
Overview
We are building a large-scale data aggregation system across 200+ listing / marketplace websites and are looking for an experienced web scraping engineer to own and maintain the scraping layer.
This is a serious, long-term project focused on reliability, cost control, and clean data.
Responsibilities
Build and maintain Scrapy spiders (one per site or site family)
Integrate Zyte API / Zyte Smart Proxy Manager
Extract structured data:
Address / location
Price (hourly / daily / monthly where applicable)
Listing URL
Handle:
Pagination / infinite scroll
JS-heavy sites (React / Vue / Angular)
XHR / JSON endpoint discovery
Export normalized JSON
Monitor and fix spiders when websites change
Optimize Zyte usage to minimize costs
Technical Requirements (Must-Have)
Strong experience with Scrapy
Experience using Zyte API or similar proxy/rendering services
Proven ability to scrape JavaScript-rendered websites
Clean, maintainable Python code
Understanding of robots.txt and ethical scraping
Git-based workflow
No Selenium-only approaches
No “browser rendering for everything”
Nice to Have
Marketplace / booking / real-estate scraping experience
XHR / API reverse-engineering skills
Cost-optimization mindset
Playwright knowledge (fallback only)
Deliverables
Scrapy spiders (clean, modular)
JSON exports per website
Configuration for:
Entry URLs
Pagination logic
Zyte usage
Basic documentation for maintenance
Project Scale
~200 websites total
Some sites: hundreds of pages
Others: tens of thousands of listings
Ongoing maintenance and expansion expected
Scrapy framework
Zyte integration
Standard JSON schema
Monthly retainer available for maintenance
We are cost-conscious and expect efficient Zyte usage.
Required Screening Questions (Do Not Skip)
In one sentence, explain the difference between Scrapy HTTP requests and browser rendering.
Name one JS-heavy website you have scraped and explain briefly how you extracted the data.
List two techniques to reduce Zyte API costs.
(Optional but recommended) Paste a short Scrapy spider skeleton (10–20 lines).
Generic answers will be rejected.
How to Apply
Please include:
Years of Scrapy experience
Example projects or GitHub (if available)
Your preferred pricing model (per-site / hourly / retainer)
Your estimated time per website
Why Work With Us?
Long-term collaboration
Clear technical direction
No shady scraping
Real production-scale system
Room to grow into a lead role