Bypass Timeout for Company Profiles Scraping
Budget: ₹1,500 – ₹12,500 INR
I'm currently scraping company profiles data from dnb.com for trend analysis, but I'm facing a timeout issue. Every time I make 4 or 5 requests per second, I get hit with a 10-second timeout.
I've already tried randomizing the request intervals to bypass this. Now, I'm looking for someone with expertise in web scraping who can help me find a solution to this issue that doesn't involve reducing the request frequency or using proxies. Your skills should ideally include:
- Advanced techniques for web scraping
- Understanding of how to handle rate limiting
- Potential use of web scraping tools or software to circumvent the issue
Your experience will be invaluable in helping me successfully gather the data I need.
eg. link: https://www.dnb.com/business-directory/company-profiles.alibaba_(china)_technology_co_ltd.e3529c2476c569b9912000545d3314d6.html
I've already tried randomizing the request intervals to bypass this. Now, I'm looking for someone with expertise in web scraping who can help me find a solution to this issue that doesn't involve reducing the request frequency or using proxies. Your skills should ideally include:
- Advanced techniques for web scraping
- Understanding of how to handle rate limiting
- Potential use of web scraping tools or software to circumvent the issue
Your experience will be invaluable in helping me successfully gather the data I need.
eg. link: https://www.dnb.com/business-directory/company-profiles.alibaba_(china)_technology_co_ltd.e3529c2476c569b9912000545d3314d6.html