Integrate MongoDB and change tracking with web scraping scripts and host them -- 2
Budget: ₹1,500 – ₹12,500 INR
I've initiated a web scraping project with a base code available on GitHub - github.com/Anurag-Baundwal/school-data-scraper-3
It consists of 4 main scripts -
1. coaches3_v2.py - for scraping college coaches data
2. r3.py - for scraping college athletics rosters - player data
3. majors_scraper.py
4. And a logos scraper
Your Key Responsibilities and Tasks:
- Modify the scripts to integrate MongoDB, ie store the output in MongoDB or MongoDB Atlas, and not just on the local filesystem (MongoDB Atlast will be preferred)
- Set up automatic weekly scraping (ie the scripts should run once per week)
- Set up change tracking - more details on this in chat
- Help host the scripts on a cloud platform like Google Cloud / AWS / Azure / Hostinger in a cost effective way. Note: Some of the scripts use up to 20 threads for faster execution.
- Run the scripts on the server and carefully check the output on the database
Ideal Skills and Experience:
- Proficiency in Python, especially in web scraping with libraries like BeautifulSoup or Scrapy (even though the scripts are already complete and scraping correctly).
- Previous experience with large scale web scraping projects.
- Ability to understand and build upon existing code.
- Experience with MongoDB Atlas
- Experience with hosting programs on cloud servers
Note: The budget for this is 2000 Rupees.
It consists of 4 main scripts -
1. coaches3_v2.py - for scraping college coaches data
2. r3.py - for scraping college athletics rosters - player data
3. majors_scraper.py
4. And a logos scraper
Your Key Responsibilities and Tasks:
- Modify the scripts to integrate MongoDB, ie store the output in MongoDB or MongoDB Atlas, and not just on the local filesystem (MongoDB Atlast will be preferred)
- Set up automatic weekly scraping (ie the scripts should run once per week)
- Set up change tracking - more details on this in chat
- Help host the scripts on a cloud platform like Google Cloud / AWS / Azure / Hostinger in a cost effective way. Note: Some of the scripts use up to 20 threads for faster execution.
- Run the scripts on the server and carefully check the output on the database
Ideal Skills and Experience:
- Proficiency in Python, especially in web scraping with libraries like BeautifulSoup or Scrapy (even though the scripts are already complete and scraping correctly).
- Previous experience with large scale web scraping projects.
- Ability to understand and build upon existing code.
- Experience with MongoDB Atlas
- Experience with hosting programs on cloud servers
Note: The budget for this is 2000 Rupees.