Modify Python Code to get around cloudflare 1020 error when scraping www.oddschecker.com
Budget: £20 – £250 GBP
Dear Freelancers,
I have some code that I have been using since July 2020 which has now stopped working.
The code will be supplied to the successful freelancer and the code will need to be modifed to fix two issues that have occured in the last two weeks.
1. The code will no longer run headless, code will run with the scraper showing in the interface. This issue started on 07 October 2021
2. The code stops running once it has scraped the first link - issue started 18 October 2021. The issue looks like once scraping loop gets to the second link, the cloudflare protection is stepped in
Skills required:
Python
BeautifulSoup
Selinium using Chromedriver
I am looking for the following:
-existing code to be fixed, not to be completely re-worked from scratch.
-as much of the original source code to be kept as orignal as possible.
-explaination of the the work done.
The solution works in the following way:
-A CSV file is used as a input for the scraper, containing URI's of oddschecker to scrape (Must not change)
-The scraper reads in the URI's and scapes (Can be changed to get around Cloudflare check)
-The output is saved to a CSV. (Must not change)
I have done some googling and believe the issue may around the user-agent being used/sent by the webdriver. I do not know how to implement this, therefore my request for help!
The successful bidder will have a long-standing relationship with me and I will come back to you for future work/enhancements.
I will provide more detail/instructions to the successful bidder.
The code may be very simple to fix, I don't care if it is a single line! - Might be a very quick job for somebody!
VPN's must not be used to get ariound the issue as the code is run on a remote server.
Thanks and Happy Bidding!
I have some code that I have been using since July 2020 which has now stopped working.
The code will be supplied to the successful freelancer and the code will need to be modifed to fix two issues that have occured in the last two weeks.
1. The code will no longer run headless, code will run with the scraper showing in the interface. This issue started on 07 October 2021
2. The code stops running once it has scraped the first link - issue started 18 October 2021. The issue looks like once scraping loop gets to the second link, the cloudflare protection is stepped in
Skills required:
Python
BeautifulSoup
Selinium using Chromedriver
I am looking for the following:
-existing code to be fixed, not to be completely re-worked from scratch.
-as much of the original source code to be kept as orignal as possible.
-explaination of the the work done.
The solution works in the following way:
-A CSV file is used as a input for the scraper, containing URI's of oddschecker to scrape (Must not change)
-The scraper reads in the URI's and scapes (Can be changed to get around Cloudflare check)
-The output is saved to a CSV. (Must not change)
I have done some googling and believe the issue may around the user-agent being used/sent by the webdriver. I do not know how to implement this, therefore my request for help!
The successful bidder will have a long-standing relationship with me and I will come back to you for future work/enhancements.
I will provide more detail/instructions to the successful bidder.
The code may be very simple to fix, I don't care if it is a single line! - Might be a very quick job for somebody!
VPN's must not be used to get ariound the issue as the code is run on a remote server.
Thanks and Happy Bidding!