Data Scraping
Budget: $10 – $30 USD
Looking to setup a repeatable process for scraping websites. Note that this solution should include Scrapy + Selenium or Splash as the websites require Javascript.
At this point the requirements are:
1) Simple scraping program based in Python using Scrapy + Selenium/Splash (Linux based)
2) Ability to configure sent request headers (rotate User Agents) and leverage rotating proxies (free) to prevent detection
3) Response Headers, Response HTML, Request Headers and Screenshot are all saved to a folder
This is just a proof of concept at this point, but will be extending this should it prove successful. No User Interface required.
At this point the requirements are:
1) Simple scraping program based in Python using Scrapy + Selenium/Splash (Linux based)
2) Ability to configure sent request headers (rotate User Agents) and leverage rotating proxies (free) to prevent detection
3) Response Headers, Response HTML, Request Headers and Screenshot are all saved to a folder
This is just a proof of concept at this point, but will be extending this should it prove successful. No User Interface required.