Experienced Python Developer Needed for Advanced Web Scraping Script
Budget: €250 – €750 EUR
Project Overview
We are seeking an experienced Python developer to create a sophisticated web scraping script. The primary goal is to extract business information from various online sources, starting with Google Maps and expanding to other platforms. Note that we only need the scraping script itself, as we have team members handling the backend, frontend, and cloud infrastructure.
Key Responsibilities
1. Develop a robust web scraping script using Python (and JavaScript if necessary) to extract business data from Google Maps.
2. Implement features to search for businesses within a specified radius of a given location.
3. Extend the scraper's capabilities to gather additional information from other online sources such as Facebook, Instagram, TikTok, and business directories.
4. Ensure the scraper can handle dynamic web content and pagination.
5. Implement data cleaning and structuring functions within the script.
6. Develop the script in a way that it can be easily integrated into our existing systems.
7. Implement error handling and logging within the script.
8. Optimize the scraper for performance and scalability.
Required Skills and Experience
- Proficient in Python programming
- Experience with JavaScript (for handling dynamic content if necessary)
- Extensive experience with web scraping libraries such as Selenium, BeautifulSoup, or Scrapy
- Strong understanding of HTML, CSS, and JavaScript
- Experience in handling dynamic web content and AJAX-based websites
- Familiarity with anti-scraping techniques and how to overcome them
- Knowledge of proxy rotation and request rate limiting
- Experience with data cleaning and structuring
- Strong problem-solving skills and attention to detail
Preferred Qualifications
- Experience scraping data from Google Maps and social media platforms
- Knowledge of legal and ethical considerations in web scraping
Project Deliverables
1. A fully functional web scraping script that can extract business information from Google Maps and other specified online sources.
2. Clean, well-documented, and maintainable code.
3. A detailed guide on how to use and maintain the script.
4. Documentation of any dependencies or setup required to run the script.
Script Requirements
- The script should be able to run independently of any specific backend or frontend.
- It should output data in a standardized format (e.g., JSON) that can be easily consumed by various systems.
- The script should be configurable via command-line arguments or a configuration file.
Timeline
The development of the script is expected to take 3-6 weeks. We may require ongoing collaboration for maintenance and feature additions based on performance.
Please include in your proposal:
1. Your relevant experience and portfolio, specifically highlighting web scraping projects
2. Your approach to developing this scraping script
3. Estimated timeline and milestones
4. Your rate and availability
We look forward to working with a skilled developer who can deliver a robust and efficient web scraping solution!
We are seeking an experienced Python developer to create a sophisticated web scraping script. The primary goal is to extract business information from various online sources, starting with Google Maps and expanding to other platforms. Note that we only need the scraping script itself, as we have team members handling the backend, frontend, and cloud infrastructure.
Key Responsibilities
1. Develop a robust web scraping script using Python (and JavaScript if necessary) to extract business data from Google Maps.
2. Implement features to search for businesses within a specified radius of a given location.
3. Extend the scraper's capabilities to gather additional information from other online sources such as Facebook, Instagram, TikTok, and business directories.
4. Ensure the scraper can handle dynamic web content and pagination.
5. Implement data cleaning and structuring functions within the script.
6. Develop the script in a way that it can be easily integrated into our existing systems.
7. Implement error handling and logging within the script.
8. Optimize the scraper for performance and scalability.
Required Skills and Experience
- Proficient in Python programming
- Experience with JavaScript (for handling dynamic content if necessary)
- Extensive experience with web scraping libraries such as Selenium, BeautifulSoup, or Scrapy
- Strong understanding of HTML, CSS, and JavaScript
- Experience in handling dynamic web content and AJAX-based websites
- Familiarity with anti-scraping techniques and how to overcome them
- Knowledge of proxy rotation and request rate limiting
- Experience with data cleaning and structuring
- Strong problem-solving skills and attention to detail
Preferred Qualifications
- Experience scraping data from Google Maps and social media platforms
- Knowledge of legal and ethical considerations in web scraping
Project Deliverables
1. A fully functional web scraping script that can extract business information from Google Maps and other specified online sources.
2. Clean, well-documented, and maintainable code.
3. A detailed guide on how to use and maintain the script.
4. Documentation of any dependencies or setup required to run the script.
Script Requirements
- The script should be able to run independently of any specific backend or frontend.
- It should output data in a standardized format (e.g., JSON) that can be easily consumed by various systems.
- The script should be configurable via command-line arguments or a configuration file.
Timeline
The development of the script is expected to take 3-6 weeks. We may require ongoing collaboration for maintenance and feature additions based on performance.
Please include in your proposal:
1. Your relevant experience and portfolio, specifically highlighting web scraping projects
2. Your approach to developing this scraping script
3. Estimated timeline and milestones
4. Your rate and availability
We look forward to working with a skilled developer who can deliver a robust and efficient web scraping solution!