python code web scraping for github

code in python web scraping for github repos

you are given link for search created from github advanced search API
like
many repos
https://github.com/cclauss
or
https://github.com/search/advanced?q=VWRegressor&type=Repositories
or
https://github.com/search?l=Python&q=lightgbm&type=Repositories
or
https://github.com/search?l=Python&q=lightgbm&type=discussions
or
https://github.com/search?l=Python&q=lightgbm&type=issues
or
https://github.com/search?q=lightgbm+categorical
or
https://github.com/search?l=Jupyter+Notebook&q=lightgbm+categorical&type=Repositories

copy past such a link to python code

download and unzip automatically all repos from this search



it should work for any link complicated from advanced search results
to downloaded all search results
like
https://github.com/search?q=Regressor++L2&type=Repositories&ref=advsearch&l=&l=

BUT DO NOT USE GITHUB API
just pure scraping as web page

unzip repos all main zip files

download repos from all folders like:

commits
python
etc

provide redownloading if repo downloading fails
prevent ban due to many downloads : for example use time delay but provide capability to download links with thousands repos like
https://github.com/cclauss

use
from selenium import webdriver
from selenium.webdriver.common.by import By
or
from requests import get

easy to install packages - teach how install needed packages

you can start from
https://github.com/Sandy4321/scarping
but make sure
1
prove repos fully downloaded - not partial zip downloading
2
redownload bad zips
3
if downgliding interrupts - windows crush or no internet crush- then start from last downloading repo ( not downloading all again )
4
prevent github ban due to many downloads
5
prove all repos downloaded from link
for example
https://github.com/Sandy4321?tab=repositories
all > 5k repos should be downloaded
6
make .txt log file