Car Parts Data Scraping System

I’m putting together a system that will regularly scrape multiple online sources for car-parts information and push the results into my product-testing database. The scraper has to collect every key data point I rely on—price, full specifications, product name, category, and any product codes—so I can run internal tests without manual look-ups.

Here’s how I picture the workflow: the script (Python with Scrapy, BeautifulSoup or a similarly reliable stack) runs on a weekly schedule, reaches the target sites, extracts the fields above, cleans obvious duplicates, and stores everything in a structured format that my current database can ingest. If an API is available for a site, feel free to use it; otherwise, a robust HTML scraper is fine as long as it respects robots.txt and keeps the scrape rate polite.

Deliverables
• A working scraper for the agreed list of car-parts sites
• Mapping of each field (price, specs, name, category, codes) to the output schema
• Automated weekly run configuration (cron, CI pipeline, or comparable scheduler)
• Simple README explaining setup, configuration, and how to add new target sites

Acceptance criteria
• Weekly job completes without errors and returns fresh data for every listed site
• All required fields populate correctly in the output file or table
• Codebase is clearly documented and easy for me to extend later

If that sounds straightforward, let me know which libraries or tooling you prefer and how soon you could deliver an initial version.