code in python web scraping for github repos

code in python web scraping for github repos

you are given link for search created from github advanced search API
like
https://github.com/search/advanced?q=VWRegressor&type=Repositories
or
https://github.com/search?q=lightgbm&type=Repositories&ref=advsearch&l=&l=

copy past such a link to python code

download and unzip all repos from this search



it should work for any link complicated from advanced search results
to downloaded all search results
like
https://github.com/search?q=Regressor++L2&type=Repositories&ref=advsearch&l=&l=

BUT DO NOT USE GITHUB API
just pure scraping as web page

unzip repos all main zip files

download repos from all folders like:

commits
python
etc

provide redownloading if repo downloading fails
prevent ban due to many downloads

use
from selenium import webdriver
from selenium.webdriver.common.by import By