web scraping for 2000 websites from wayback machine
Budget: $30 – $250 USD
I want you to scrape for me 2000 different websites that are archived on wayback machine.
where I have there exact urls.
such as: https://web.archive.org/web/20150206061758/http://www.m-hikari.com/ces/index.html
if you are willing to use scrapy .. I have the full code and I can give it to you and explain it to you.
however it is slow because web.archive.org do not respond for rvery request.
if you are want to use requests and beutiful soup, then I have a code that can do all my requierments on any website except it does not deal with the wayback machine.
here is my requierments if you want to build the code from zero:
1- I need the website textual content of 3,000 (not HTML I need the text) up to level\depth (3) for every website.
2- include the dynamic content of the website produced by Javascript
3- include the textual content of downloadable files such as PDF and word documents
4- example output is atached. where if the type is a file I did not attach the content but I downlod it on computer.
where I have there exact urls.
such as: https://web.archive.org/web/20150206061758/http://www.m-hikari.com/ces/index.html
if you are willing to use scrapy .. I have the full code and I can give it to you and explain it to you.
however it is slow because web.archive.org do not respond for rvery request.
if you are want to use requests and beutiful soup, then I have a code that can do all my requierments on any website except it does not deal with the wayback machine.
here is my requierments if you want to build the code from zero:
1- I need the website textual content of 3,000 (not HTML I need the text) up to level\depth (3) for every website.
2- include the dynamic content of the website produced by Javascript
3- include the textual content of downloadable files such as PDF and word documents
4- example output is atached. where if the type is a file I did not attach the content but I downlod it on computer.