Urgent Python Script Fix for Web Crawling using scrapy or botasaurus
Budget: $30 – $250 USD
i have a python script that i need to add the following features.
1. use scrapy or botasaurus
2. handle errors
3. captcha bypass
4. correct proxy handling
5. fix script to start crawling w new website
it has currently these features
1. I need to crawl any website.
2. Create a CSV file sitemap with parent URL and all children URLs in each parent.
3. Indicate if the child URL is an internal or external link.
4. Do not crawl external links.
5. Indicate the status of the parent URL (200, 404, etc).
6. Use cache.
7. Use proxy rotation.
8. Use multiple browsers simultaneously to make the crawling fast
9. use pyppetter / selenium / or anything similar to it to be able to read java based websites
11. implement script continue where it left off
12. implement make a second copy of the cache to html2text
13. implement batch size to save in memory the data until batch size is reached then data will be written to disk (to make script faster)
14. proxy retry if fails
15. if proxy was found bad while downloading change proxy and try again until success
16. error handeling
17. do not use sitemap.xml file but crawl and create your own csv file
18. handle any interruption
19. color code messages
20. make the script undetectable as a bot
you can test the script on this website
https://www.inautia.com/boats/condition-used/type-sailboats/
88-digital.com
please check project files for more information
1. use scrapy or botasaurus
2. handle errors
3. captcha bypass
4. correct proxy handling
5. fix script to start crawling w new website
it has currently these features
1. I need to crawl any website.
2. Create a CSV file sitemap with parent URL and all children URLs in each parent.
3. Indicate if the child URL is an internal or external link.
4. Do not crawl external links.
5. Indicate the status of the parent URL (200, 404, etc).
6. Use cache.
7. Use proxy rotation.
8. Use multiple browsers simultaneously to make the crawling fast
9. use pyppetter / selenium / or anything similar to it to be able to read java based websites
11. implement script continue where it left off
12. implement make a second copy of the cache to html2text
13. implement batch size to save in memory the data until batch size is reached then data will be written to disk (to make script faster)
14. proxy retry if fails
15. if proxy was found bad while downloading change proxy and try again until success
16. error handeling
17. do not use sitemap.xml file but crawl and create your own csv file
18. handle any interruption
19. color code messages
20. make the script undetectable as a bot
you can test the script on this website
https://www.inautia.com/boats/condition-used/type-sailboats/
88-digital.com
please check project files for more information