Python Data Scraping Automation Script

I need a self-contained Python 3 script that automatically scrapes data from a set of webpages, cleans it, and saves the results in a structured format I can import into other tools. The workflow should be completely hands-off once configured: the script fetches each target URL, follows any pagination or internal links I specify, extracts the required fields, and writes everything to CSV (JSON as a secondary option would be helpful too).

Please build it so I can adjust targets and CSS/XPath selectors without touching the core logic—ideally through a simple config section or external file. Friendly error handling, basic logging, and compliance with robots.txt are important because this will run on a schedule. I’m happy for you to lean on libraries such as requests, BeautifulSoup, Scrapy, or Selenium where appropriate; choose whatever keeps the code readable and the scrape reliable.

Deliverables:
• Clean, well-commented Python source code
• A short README covering setup, dependencies, and usage examples
• One sample output file showing the data structure

Acceptance criteria: when I execute the script on my machine, it retrieves the agreed-upon fields from the test site, generates the sample file without errors, and logs each step of the process clearly.