Website Data Mining // Web Scraping // Database
Budget: $30 – $250 USD
Project Scope & Needs
1. See the linked spreadsheet below for a list of all URLs and data needed to be scraped (there are multiple tabs - each tab must be complete).
2. Data Needs
> (a) every undergraduate majors for each university and college listed. URL is listed in column AY. Majors should be listed in columns AZ - QI.
> (b) College Softball Coaches by school. Data needed. Head Coach Name, Head Coach Title, Head Coach email address, Head Coach Phone #, Head Coach Twitter/X handle. I will need the exact same data for all Assistant Coaches (where available), Director of Operations and Recruiting Coordinator. URL for each school is column V. Data can go into columns W - AX.
> (c) College Softball Roster URLs are listed in column U. You will need to create new columns for this data, but here is what I need to capture. Data needed - roster year (2024, 2025, etc), every player's name, every player's position, every player's year (FY, So, Jr, Sr, Grad, etc), and Every player's hometown. We will need a way in the database to convert the players year into a graduation year. For example, a 2024 roster, a Grad and Sr will graduate in 2024, Jr 2025, So 2026, FR or FY 2027. A 2025 roster, Grad / Sr will be 2025 Grad, Jr 2026, So 2027, Fr/Fy 2028. We will need this to be able to continue update as we add more roster years in the future.
> (D) The athletic logos for each university or college listed. This just needs to be stored in the database. The list of athletic website URLs to pull logo from is in column T. This is a one time pull to get logo into the database.
**If any links are not the correct page, I will need to know to provide an update**
3. Add the data to the google sheet first, then go to next step
4. Build a database that will seamlessly work with an e-commerce website platform (the website is under construction) but this database will need to work with the site on the backend and then on the frontend for a customer search tool
**6. Build a web scraper tool to update the data in the database on a weekly basis - leveraging the 3 URLs per school and the data mentioned in #2 above needs to be part of this scope. This would need to be automated when it moves to the server and website in the future.
> Needs Bypass Protections: Implement mechanisms to bypass IP restrictions, Captcha, and anti-bot or cloudflare protections.
7. Change Tracking
> Database Monitoring: Implement functionality to track changes in the database over time with weekly web scraping updates.
> Change Reports: Generate reports highlighting changes (e.g., new coach hires) for validation and update purposes.
Timing ASAP - need best pricing as I am on a very tight budget
As mentioned the website is under construction. Will need a recommendation for a cheap hosting solution to house the database and the website. Requirements: cheap monthly, fast and responsive, secure.
** You will need to move the database and automated scraping tool to my server and ensure there are no issues**
If the project goes well, will have additional projects for the successful bidder.
LINK TO SPREADSHEET:
https://docs.google.com/spreadsheets/d/1rX7WxUbMfn95GsmfCMdvE9AGQ96NJIa6aWALYRa1v3c/edit?usp=sharing
1. See the linked spreadsheet below for a list of all URLs and data needed to be scraped (there are multiple tabs - each tab must be complete).
2. Data Needs
> (a) every undergraduate majors for each university and college listed. URL is listed in column AY. Majors should be listed in columns AZ - QI.
> (b) College Softball Coaches by school. Data needed. Head Coach Name, Head Coach Title, Head Coach email address, Head Coach Phone #, Head Coach Twitter/X handle. I will need the exact same data for all Assistant Coaches (where available), Director of Operations and Recruiting Coordinator. URL for each school is column V. Data can go into columns W - AX.
> (c) College Softball Roster URLs are listed in column U. You will need to create new columns for this data, but here is what I need to capture. Data needed - roster year (2024, 2025, etc), every player's name, every player's position, every player's year (FY, So, Jr, Sr, Grad, etc), and Every player's hometown. We will need a way in the database to convert the players year into a graduation year. For example, a 2024 roster, a Grad and Sr will graduate in 2024, Jr 2025, So 2026, FR or FY 2027. A 2025 roster, Grad / Sr will be 2025 Grad, Jr 2026, So 2027, Fr/Fy 2028. We will need this to be able to continue update as we add more roster years in the future.
> (D) The athletic logos for each university or college listed. This just needs to be stored in the database. The list of athletic website URLs to pull logo from is in column T. This is a one time pull to get logo into the database.
**If any links are not the correct page, I will need to know to provide an update**
3. Add the data to the google sheet first, then go to next step
4. Build a database that will seamlessly work with an e-commerce website platform (the website is under construction) but this database will need to work with the site on the backend and then on the frontend for a customer search tool
**6. Build a web scraper tool to update the data in the database on a weekly basis - leveraging the 3 URLs per school and the data mentioned in #2 above needs to be part of this scope. This would need to be automated when it moves to the server and website in the future.
> Needs Bypass Protections: Implement mechanisms to bypass IP restrictions, Captcha, and anti-bot or cloudflare protections.
7. Change Tracking
> Database Monitoring: Implement functionality to track changes in the database over time with weekly web scraping updates.
> Change Reports: Generate reports highlighting changes (e.g., new coach hires) for validation and update purposes.
Timing ASAP - need best pricing as I am on a very tight budget
As mentioned the website is under construction. Will need a recommendation for a cheap hosting solution to house the database and the website. Requirements: cheap monthly, fast and responsive, secure.
** You will need to move the database and automated scraping tool to my server and ensure there are no issues**
If the project goes well, will have additional projects for the successful bidder.
LINK TO SPREADSHEET:
https://docs.google.com/spreadsheets/d/1rX7WxUbMfn95GsmfCMdvE9AGQ96NJIa6aWALYRa1v3c/edit?usp=sharing