Python GitHub User Scraper
Budget: $30 – $250 USD
I need a Python-based scraper that targets GitHub Users specifically. The tool must let me specify three filters before each run—Location, Keyword (bio or repo-related), and minimum Number of followers—then crawl every matching profile.
Key capabilities I expect
• Robust pause / resume: if I stop the script mid-run it should restart later from the exact last profile processed.
• Field capture: email (when public), full name, username, profile URL, location, personal website link, followers count, and any other easily accessible profile details.
• Clean Excel export: once the crawl finishes, everything is written to a single .xlsx file with clear column headers.
• Rate-limit awareness: the code must respect GitHub’s API or HTML limits, back off automatically, and allow me to supply personal access tokens when needed.
Deliverables
1. A well-commented Python 3 script (or small module) ready to run from the command line.
2. requirements.txt with all libraries, preferably relying on popular requests, BeautifulSoup, Selenium, or the official GitHub API—whatever you find most stable.
3. README.md explaining setup, how to set filters, how the checkpoint / resume mechanism works, and an example command.
4. A sample Excel output generated from a short demo run for verification.
Acceptance criteria
– The scraper returns accurate data for at least 100 test users matching given filters.
– Stopping the process and relaunching reproduces identical results without duplicates or gaps.
– Excel contains all requested columns in the correct order.
Keep the code lightweight, self-contained, and ready for me to extend later. Let me know if you foresee any hurdles with GitHub’s policies so we can address them early. I’m ready to start as soon as you confirm the approach.
Key capabilities I expect
• Robust pause / resume: if I stop the script mid-run it should restart later from the exact last profile processed.
• Field capture: email (when public), full name, username, profile URL, location, personal website link, followers count, and any other easily accessible profile details.
• Clean Excel export: once the crawl finishes, everything is written to a single .xlsx file with clear column headers.
• Rate-limit awareness: the code must respect GitHub’s API or HTML limits, back off automatically, and allow me to supply personal access tokens when needed.
Deliverables
1. A well-commented Python 3 script (or small module) ready to run from the command line.
2. requirements.txt with all libraries, preferably relying on popular requests, BeautifulSoup, Selenium, or the official GitHub API—whatever you find most stable.
3. README.md explaining setup, how to set filters, how the checkpoint / resume mechanism works, and an example command.
4. A sample Excel output generated from a short demo run for verification.
Acceptance criteria
– The scraper returns accurate data for at least 100 test users matching given filters.
– Stopping the process and relaunching reproduces identical results without duplicates or gaps.
– Excel contains all requested columns in the correct order.
Keep the code lightweight, self-contained, and ready for me to extend later. Let me know if you foresee any hurdles with GitHub’s policies so we can address them early. I’m ready to start as soon as you confirm the approach.
Related categories:
Python
Excel
Web Scraping
OAuth
Web Development
Data Analysis
Selenium
Automation
API Integration
Pandas