Business Directory Data Scraping

I have access to an online business directory containing roughly 30,000 company profiles. I need every publicly visible field on each profile—think phone numbers, email addresses, physical addresses, descriptions, website links and any other details the page exposes—captured and delivered in a single Google Sheets file.

Please build or run an automated scraper (Python + BeautifulSoup, Scrapy, Selenium, or a comparable stack) that can:

• Crawl every profile, including deeper pages reached via pagination or “load more” buttons.
• Respect the site’s structure and timing so we stay under any rate-limit or anti-bot radar.
• Deduplicate records and keep data clean (no broken lines, hidden HTML tags, or merged cells).
• Push the final dataset directly into a shareable Google Sheet, preserving column headers that mirror the field names found on the site.

Deliverables I expect:
1. The populated Google Sheet with ~30 k complete, clean rows.
2. The runnable script and brief setup notes so I can re-execute the crawl later if needed.

I’ll provide the directory URL and any login or search parameters as soon as we start. Let me know your estimated turnaround time and any questions about site structure so you can hit the ground running.