Apify Web Scraping Robot Setup
Budget: $3,000 – $5,000 USD
I need a custom Apify actor that reliably scrapes text data from a list of websites I will supply. The robot should:
• Visit each URL, respect robots.txt and crawl-delay rules, and extract the specific text blocks I flag (headings, paragraphs, metadata, or any structured selectors I provide).
• Store results in an Apify dataset in JSON/CSV format ready for download or API access.
• Handle pagination and basic anti-bot measures (rotating proxies, session management, retries).
• Log progress clearly so I can track run status inside the Apify console.
Please build the actor in JavaScript/TypeScript, include a concise README with setup, environment variables, and an example schedule. Once finished, share the Apify actor link or GitHub repo so I can test the run on my own account.
• Visit each URL, respect robots.txt and crawl-delay rules, and extract the specific text blocks I flag (headings, paragraphs, metadata, or any structured selectors I provide).
• Store results in an Apify dataset in JSON/CSV format ready for download or API access.
• Handle pagination and basic anti-bot measures (rotating proxies, session management, retries).
• Log progress clearly so I can track run status inside the Apify console.
Please build the actor in JavaScript/TypeScript, include a concise README with setup, environment variables, and an example schedule. Once finished, share the Apify actor link or GitHub repo so I can test the run on my own account.