Web Scraping Specialist for Insightful Content

Title:
Senior Web Scraping & Data Engineering Developer for Content Intelligence Platform

Project Overview:
I am building a content‑aggregation and insights pipeline that collects updates, posts, and signals from major online platforms and public sources. The output will support a monthly insights publication and a long‑term analytics framework. I am seeking an experienced developer from a cost‑efficient region with strong skills in web scraping, browser automation, data engineering, and API integration.

Key Responsibilities:

Build and maintain scraping workflows for platforms such as LinkedIn, X/Twitter, Reddit, Substack, Facebook, Instagram, and other public sources

Implement browser-based automation using tools like Playwright, Puppeteer, or Selenium

Use lightweight scrapers where appropriate to reduce compute usage

Create profile-based, organization-based, and keyword-based scraping approaches

Optimize concurrency, session handling, and anti-blocking strategies

Produce structured data outputs (JSON/CSV/DB) for downstream analytics

Implement delta scraping to capture only new content efficiently

Integrate with Apify Actors or custom Node.js/Python scripts

Maintain reliability as platforms evolve or change their anti-bot systems

Required Skills:

Strong experience with Python or Node.js

Expertise in browser automation frameworks such as Playwright, Puppeteer, or Selenium

Experience with Apify, Scrapy, or similar scraping frameworks

Knowledge of proxy rotation, session management, and anti-detection techniques

Ability to design scalable, maintainable scraping pipelines

Familiarity with REST APIs, OAuth, and data normalization

Experience with MongoDB, PostgreSQL, or cloud storage solutions

Preferred Experience:

Prior work scraping platforms with strong anti-bot protections

Experience building content aggregation, monitoring, or intelligence systems

Familiarity with summarization, tagging, or enrichment workflows (nice to have)

Engagement Details:

Long-term engagement (3–12 months)

Weekly deliverables

Competitive compensation based on experience

Must be available for periodic calls in US Eastern Time evenings

To Apply:
Please share:

Examples of similar scraping or automation projects

Your preferred tech stack

Your experience with browser automation

Your availability and hourly/monthly rate