Listings Web Scraping Automation

I need a repeatable script—written in either Python (Requests / BeautifulSoup, Scrapy, or Selenium) or JavaScript (Node.js with Puppeteer or Cheerio)—that automatically collects listing data from a set of niche websites. These sites fall into the “other” category; they are neither e-commerce, news, nor social-media platforms, yet each publishes publicly accessible listings across multiple paginated pages.

What the scraper must do:
• Traverse every page (following pagination) of each target site.
• Extract all key fields from every listing: title, URL, posted date, description snippet, plus any price or ID field present.
• De-duplicate results so reruns only capture new listings.
• Save the output to both CSV and JSON files; a simple SQLite dump would be a welcome bonus.
• Run headless so it can be scheduled from a cron job or Windows Task Scheduler.

Delivery expectations:
• Clean, well-commented code with selectors and target URLs stored in an easily editable config.
• A concise README that details setup, required libraries, and how to extend the scraper to new sites.
• A quick demo (screenshots or short clip) showing successful extraction from at least one target site so I can verify everything works before final sign-off.

Open-source libraries are fine as long as their licences allow commercial use. Once the scraper is stable, I’ll likely move to a second milestone for Dockerisation and cloud deployment, so please code with portability in mind.

Let me know which language you prefer and how soon you can deliver a first working version.