Python Web Scraper - Extend what we've already built
Budget: $30 – $250 USD
Project Description: Find school districts and charter schools who use a specific vendor for a large list of domains.
I am seeking an experienced web scraping specialist to improve our Python script to analyze a large list of school district websites (approximately 4000+ URLs) and identify the ones who show a specific link on any page found in their sitemap.
The primary method of identification must be to scan the website's sitemap.xml for specific, known vendor links.
Deliverables Required
1. A Production-Ready Python Script (.py file): The script must be commented, easily configurable, and capable of reading the provided CSV list, performing the scan, and generating the output CSV. It should handle timeouts and basic error handling gracefully.
2. The Final Results (CSV/Excel File): A clean data file containing the results for all URLs provided. The resulting CSV should include original variables like organization name, state and zip even though that data was not used in the scraper.
The script must perform the following steps for each URL in the input list:
1. Input: Read a list of URLs from a provided CSV file (single column of URLs).
2. Navigation/Rendering: Visit the URL (handling redirects is essential). The use of a headless browser (like Selenium/Puppeteer) or an advanced HTTP library is preferred, as some websites may load the footer content dynamically via JavaScript.
3. Targeted Scanning: Scan the HTML source code of all pages found in the sitemap, specifically looking for the presence of a specific link.
4. Output Logic:
- If the link is found, record the identified vendor.
- If no vendor is explicitly identified, the output should list the status as `"No Vendor Found"`.
- If no website could be loaded, the script should log any failed connections or timeouts.
Output Format (CSV)
The final deliverable file should be structured with the same columns as the ones provided with the additional column to include your results.
Skills Required
- Expert proficiency in Python.
- Deep experience with web scraping libraries (e.g., Requests, BeautifulSoup, Scrapy, and especially Selenium/Puppeteer for dynamic content).
- Experience handling common web scraping challenges (redirects, user-agents, proxy usage (if necessary)).
To bid, please confirm your familiarity with scraping dynamic content and provide a brief description of the scraping approach you would use.
I am seeking an experienced web scraping specialist to improve our Python script to analyze a large list of school district websites (approximately 4000+ URLs) and identify the ones who show a specific link on any page found in their sitemap.
The primary method of identification must be to scan the website's sitemap.xml for specific, known vendor links.
Deliverables Required
1. A Production-Ready Python Script (.py file): The script must be commented, easily configurable, and capable of reading the provided CSV list, performing the scan, and generating the output CSV. It should handle timeouts and basic error handling gracefully.
2. The Final Results (CSV/Excel File): A clean data file containing the results for all URLs provided. The resulting CSV should include original variables like organization name, state and zip even though that data was not used in the scraper.
The script must perform the following steps for each URL in the input list:
1. Input: Read a list of URLs from a provided CSV file (single column of URLs).
2. Navigation/Rendering: Visit the URL (handling redirects is essential). The use of a headless browser (like Selenium/Puppeteer) or an advanced HTTP library is preferred, as some websites may load the footer content dynamically via JavaScript.
3. Targeted Scanning: Scan the HTML source code of all pages found in the sitemap, specifically looking for the presence of a specific link.
4. Output Logic:
- If the link is found, record the identified vendor.
- If no vendor is explicitly identified, the output should list the status as `"No Vendor Found"`.
- If no website could be loaded, the script should log any failed connections or timeouts.
Output Format (CSV)
The final deliverable file should be structured with the same columns as the ones provided with the additional column to include your results.
Skills Required
- Expert proficiency in Python.
- Deep experience with web scraping libraries (e.g., Requests, BeautifulSoup, Scrapy, and especially Selenium/Puppeteer for dynamic content).
- Experience handling common web scraping challenges (redirects, user-agents, proxy usage (if necessary)).
To bid, please confirm your familiarity with scraping dynamic content and provide a brief description of the scraping approach you would use.
Related categories:
Python
Web Scraping
Django
Software Architecture
Scrapy
Data Extraction
BeautifulSoup
Data Analysis
Selenium
Automation