use proxy server for VB.NET data scraping program
Budget: $30 – $250 USD
I have a program written in VB.NET that scrapes data from a public site. The program submits a case number to the site, the site looks up the case, and then my program stores the html returned by the site into an Access database.
My problem is that after looking up a few hundred cases, the site realizes my program is a bot, and it stops serving the pages to me.
My proposed solution is to use a proxy server (zyte.com?) to use different IP addresses when scraping the site. Also, I think I should randomize the order of cases that I process. For example instead of scraping the data for ID 101V, then 102V, then 103V, 104V, 105V,106V, 107V, 108V, 109V, 110V, I think I should scrape the data in a random order like: 108V, 102V, 105V,110V, 101V, 107V, 109Vv, 104V, 103V, 106V.
I think this is an easy project. I just don’t have time to do it.
I can pay you $100 to do this project. And I will pay for the cost of proxy server.
Please let me know when you could have this done by. If your proposal looks ok, I will send you the VB.NET code and the Access database.
Thanks.
My problem is that after looking up a few hundred cases, the site realizes my program is a bot, and it stops serving the pages to me.
My proposed solution is to use a proxy server (zyte.com?) to use different IP addresses when scraping the site. Also, I think I should randomize the order of cases that I process. For example instead of scraping the data for ID 101V, then 102V, then 103V, 104V, 105V,106V, 107V, 108V, 109V, 110V, I think I should scrape the data in a random order like: 108V, 102V, 105V,110V, 101V, 107V, 109Vv, 104V, 103V, 106V.
I think this is an easy project. I just don’t have time to do it.
I can pay you $100 to do this project. And I will pay for the cost of proxy server.
Please let me know when you could have this done by. If your proposal looks ok, I will send you the VB.NET code and the Access database.
Thanks.