Python Text Scraper Development - 06/06/2026 11:40 EDT

I need a clean, well-commented Python script that starts from a single root URL, follows every internal link it finds (no keyword or structural filtering at all), and extracts only the visible text content on each page.

The script should rely on mainstream libraries—requests plus BeautifulSoup is fine, but feel free to propose Scrapy or an async stack if it fits better. Please keep the code modular so I can later drop individual functions into a bigger application.

Core expectations
• Crawl every reachable link within the domain, respecting robots.txt and an adjustable polite delay.
• Skip images, PDFs, or other binary assets; focus strictly on textual information.
• Save each page’s URL alongside the extracted text in a single output file (CSV or JSON—whichever you prefer is acceptable).
• Handle time-outs, redirects, and JavaScript-heavy pages gracefully so the run never crashes halfway through.
• Include a short README that covers environment setup, command to launch the crawl, and tunable arguments such as depth and rate-limit.

Acceptance test
Running `python scraper.py https://example.com` on my machine must finish without uncaught exceptions and produce the output file containing every crawled URL plus its text.

That’s the whole task—once the script meets the above criteria, the project is complete.