Improve my web-scraper made from Scrapy in Python. Get passed restrictions from the site (ip-ban).
Budget: $30 – $250 USD
Im a mobile developer with some Python experience. So I made a web-scraper in Python with the Scrapy framework. The spider scrapes every fighter from Tapology.com and the information about them and all the fights they have fought. To do this I have a list of links with all the fighters separated by their country (for example the link for Norwegian fighters are: https://www.tapology.com/search/mma-fighters-by-nationality/country-no). For each country there are profiles for each fighter in that country. For example, the first fighter in the Norwegian list is https://www.tapology.com/fightcenter/fighters/100362-mohammed . If you go to his profile you can see the Fighter Details about him (height, age, record, country and so on) and below you have his Amateur Results. Some fighters have only Amateur results, some have only Pro Results, some have both of these and some have none. The scraper already takes this into consideration. The spider scrapes the Fighter Details normally, but to get the results below I have to use the API-endpoint of the site.
The first and biggest problem is that after I have been scraping for a while, Tapology blocks my IP even though I use free proxy provider from Github and fake proxy headers from Github. I manage to scrape again with VPN service, but after a while they block that IP too. The VPN doesn't have a rotating solution. I will pay extra if you can get passed the IP-ban, if not I can use professional proxies, but I prefer not to.
The second problem is that the scraper has to be as fast and reliable as possible, since Im scraping a lot of data and it have to do it multiple times. So I haven´t done any error-handling or made multiple requests at the same time. So it would be great if you can fix this.
The third problem is the JSON that I get when I scrape. First I haven`t handled null values, n\, \n or \nN/A\n. The second is the structure of the JSON. I get all the data I need, but I also get data that I don´t need. In the pro_record and amateur_record array each fight the fighter has fought is saved. For each fight in the arrays I want to remove all the data from the child called info except the child of info called attributes. Then I want attributes to replace info. (see screenshot for JSON_example)
The source code is in the description. The JSON-file is the last scrape I did from the Swedish fighters. It isn't the full data since my IP got blocked. You can also delete comments if you like to.
The first and biggest problem is that after I have been scraping for a while, Tapology blocks my IP even though I use free proxy provider from Github and fake proxy headers from Github. I manage to scrape again with VPN service, but after a while they block that IP too. The VPN doesn't have a rotating solution. I will pay extra if you can get passed the IP-ban, if not I can use professional proxies, but I prefer not to.
The second problem is that the scraper has to be as fast and reliable as possible, since Im scraping a lot of data and it have to do it multiple times. So I haven´t done any error-handling or made multiple requests at the same time. So it would be great if you can fix this.
The third problem is the JSON that I get when I scrape. First I haven`t handled null values, n\, \n or \nN/A\n. The second is the structure of the JSON. I get all the data I need, but I also get data that I don´t need. In the pro_record and amateur_record array each fight the fighter has fought is saved. For each fight in the arrays I want to remove all the data from the child called info except the child of info called attributes. Then I want attributes to replace info. (see screenshot for JSON_example)
The source code is in the description. The JSON-file is the last scrape I did from the Swedish fighters. It isn't the full data since my IP got blocked. You can also delete comments if you like to.