python scraping github repos

code in python web scraping for github repos

you are given link for search created from github advanced search API
like
many repos
https://github.com/cclauss
or
https://github.com/search/advanced?q=VWRegressor&type=Repositories
or
https://github.com/search?q=lightgbm&type=Repositories&ref=advsearch&l=&l=

copy past such a link to python code

download and unzip all repos from this search



it should work for any link complicated from advanced search results
to downloaded all search results
like
https://github.com/search?q=Regressor++L2&type=Repositories&ref=advsearch&l=&l=

BUT DO NOT USE GITHUB API
just pure scraping as web page

unzip repos all main zip files

download repos from all folders like:

commits
python
etc

provide redownloading if repo downloading fails
prevent ban due to many downloads : for example use time delay but provide capability to download links with thousands repos like
https://github.com/cclauss

use
from selenium import webdriver
from selenium.webdriver.common.by import By
or
from requests import get

you can start from
https://github.com/Sandy4321/scarping
but make sure
1
prove repos fully downloaded - not partial zip downloading
2
redownload bad zips
3
if downgliding interrupts - windows crush or no internet crush- then start from last downloading repo ( not downloading all again )
4
prevent github ban due to many downloads
5
prove all repos downloaded from link
for example
https://github.com/Sandy4321?tab=repositories
all > 5k repos should be downloaded