Fix problem with connection in Scrapy project in python
Budget: €30 – €250 EUR
The job would consist of reviewing an existing web scraping project in relation to how the connection to a proxy is being made in order to avoid detection by the visited page. The project is made in Scrapy and currently, despite the fact that the proxy IP changes with each connection and has around 700,000 IPs, we are still being detected. In the settings.py file, these are the values that exist for variables that may have something to do with the problem posed:
DOWNLOAD_TIMEOUT = 5
RETRY_TIMES = 200
CONCURRENT_REQUESTS = 10
Wheh we extract the data we do it in two steps:
Level1: we catch all the urls where we are going to get the info.
Level2. We visit each unrl in level1 and we catch all the data we want.
For both levels you must fix the same problem with the connection.
I send a Python file with the file for the spider (it’s not complete, but in this file you can see the part you’d have to touch with the connection).
If you have read the project write UNDERSTOOD in your answer.
DOWNLOAD_TIMEOUT = 5
RETRY_TIMES = 200
CONCURRENT_REQUESTS = 10
Wheh we extract the data we do it in two steps:
Level1: we catch all the urls where we are going to get the info.
Level2. We visit each unrl in level1 and we catch all the data we want.
For both levels you must fix the same problem with the connection.
I send a Python file with the file for the spider (it’s not complete, but in this file you can see the part you’d have to touch with the connection).
If you have read the project write UNDERSTOOD in your answer.