Custom Python Web Scraper to identify website hosting provider
Budget: $100 – $300 USD
Project Description: Extract Website Vendor/CMS for Large List domains.
I am seeking an experienced web scraping specialist to build and run a reliable Python script to analyze a large list of school district websites (approximately 4000+ URLs) and identify the specific hosting/CMS vendor used for each site.
The primary method of identification must be to scan the website's **footer HTML** for specific, known vendor signature strings. The project goal is a clean, automated deliverable suitable for ongoing use.
Deliverables Required
1. A Production-Ready Python Script (.py file): The script must be commented, easily configurable, and capable of reading the provided CSV list, performing the scan, and generating the output CSV. It should handle timeouts and basic error handling gracefully.
2. The Final Results (CSV/Excel File): A clean data file containing the results for all URLs provided. The resulting CSV should include original variables like organization name, state and zip even though that data was not used in the scraper.
The script must perform the following steps for each URL in the input list:
1. Input: Read a list of URLs from a provided CSV file (single column of URLs).
2. Navigation/Rendering: Visit the URL (handling redirects is essential). The use of a headless browser (like Selenium/Puppeteer) or an advanced HTTP library is preferred, as some websites may load the footer content dynamically via JavaScript.
3. Targeted Scanning: Scan the HTML source code, specifically looking for the presence of a set of vendor signature strings to be provided when the job is awarded.
4. Output Logic:
- If any signature string is found, record the identified vendor.
- If no vendor is explicitly identified, the output should list the status as `"No Vendor Found"`.
- If no website could be loaded, the script should log any failed connections or timeouts.
Output Format (CSV)
The final deliverable file should be structured with the same columns as the ones provided with the additional column to include your results.
Skills Required
- Expert proficiency in Python.
- Deep experience with web scraping libraries (e.g., Requests, BeautifulSoup, Scrapy, and especially Selenium/Puppeteer for dynamic content).
- Experience handling common web scraping challenges (redirects, user-agents, proxy usage (if necessary)).
To bid, please confirm your familiarity with scraping dynamic content and provide a brief description of the scraping approach you would use.
I am seeking an experienced web scraping specialist to build and run a reliable Python script to analyze a large list of school district websites (approximately 4000+ URLs) and identify the specific hosting/CMS vendor used for each site.
The primary method of identification must be to scan the website's **footer HTML** for specific, known vendor signature strings. The project goal is a clean, automated deliverable suitable for ongoing use.
Deliverables Required
1. A Production-Ready Python Script (.py file): The script must be commented, easily configurable, and capable of reading the provided CSV list, performing the scan, and generating the output CSV. It should handle timeouts and basic error handling gracefully.
2. The Final Results (CSV/Excel File): A clean data file containing the results for all URLs provided. The resulting CSV should include original variables like organization name, state and zip even though that data was not used in the scraper.
The script must perform the following steps for each URL in the input list:
1. Input: Read a list of URLs from a provided CSV file (single column of URLs).
2. Navigation/Rendering: Visit the URL (handling redirects is essential). The use of a headless browser (like Selenium/Puppeteer) or an advanced HTTP library is preferred, as some websites may load the footer content dynamically via JavaScript.
3. Targeted Scanning: Scan the HTML source code, specifically looking for the presence of a set of vendor signature strings to be provided when the job is awarded.
4. Output Logic:
- If any signature string is found, record the identified vendor.
- If no vendor is explicitly identified, the output should list the status as `"No Vendor Found"`.
- If no website could be loaded, the script should log any failed connections or timeouts.
Output Format (CSV)
The final deliverable file should be structured with the same columns as the ones provided with the additional column to include your results.
Skills Required
- Expert proficiency in Python.
- Deep experience with web scraping libraries (e.g., Requests, BeautifulSoup, Scrapy, and especially Selenium/Puppeteer for dynamic content).
- Experience handling common web scraping challenges (redirects, user-agents, proxy usage (if necessary)).
To bid, please confirm your familiarity with scraping dynamic content and provide a brief description of the scraping approach you would use.
Related categories:
Python
Web Scraping
Django
Software Architecture
Scrapy
Data Extraction
BeautifulSoup
Data Analysis
Selenium
Automation