Advanced Private Web Scraper Development

Project Title: Private Advanced Web Scraper with Live Updates

Description:
I need a custom-built web scraper for my personal use. This isn’t for resale or distribution — it’s a private tool that I’ll be the only one using.

What I’m looking for:

Config-based scraper: I should be able to adjust a JSON config to target different sites (CSS selectors for fields, pagination, next button, etc.).

Live status logging: Console updates + simple log file so I can watch progress while it runs.

Checkpointing: If the scraper stops, I should be able to resume without starting over.

Deduplication: No duplicate items.

Output formats: Save results continuously to JSONL and easily convert/export to CSV.

Polite scraping: Built-in delays, retries, user-agent handling.

Optional: A simple HTML status page that auto-refreshes so I can track progress visually.

Deliverables:

Main scraper script (Python preferred, Requests + BeautifulSoup or Selenium if needed)

Example config file so I can swap in my own targets

Helper to convert JSONL → CSV

README for setup and usage

Timeline: By Monday