Fix Proxy Rotation & Scraping in Chrome Extension
Budget: $30 – $250 USD
Chrome Extension Developer Needed to Fix Proxy Rotation and Data Scraping Issues
**Project Description:**
I have a Chrome extension designed to scrape data from TruePeopleSearch.com. However, I'm encountering two primary issues that need to be resolved:
1. **Proxy Rotation:** The extension is not properly rotating through the provided list of proxies. This is resulting in IP blocks and preventing the scraper from running effectively. I need a developer to implement a robust proxy rotation mechanism that ensures each request is made through a different proxy from the list. The proxy list is provided in a CSV file. The extension currently supports HTTP proxies with username/password authentication.
2. **407 Proxy Authentication Error:** The extension is encountering a "407 Proxy Authentication Required" error, indicating that the proxy authentication is not being handled correctly. I need a developer to troubleshoot and fix the proxy authentication process to ensure the extension can successfully authenticate with the proxies.
3. **CAPTCHA Handling:** The extension needs to automatically detect CAPTCHA pages on TruePeopleSearch.com and pause the scraping process, allowing the user to manually solve the CAPTCHA.
4. **Automated Scraping:** The extension needs to automatically click the "View Details" button on the search results page and scrape the data from the contact details page.
**Specific Requirements:**
- **Proxy Rotation:**
- Implement a reliable proxy rotation mechanism using a list of HTTP proxies from a CSV file.
- Ensure proper handling of username/password authentication for proxies.
- Implement error handling for failed proxy connections and automatic retries with a different proxy.
- **407 Proxy Authentication Error:**
- Identify and resolve the cause of the 407 error.
- Ensure the extension correctly authenticates with the proxy server.
- **CAPTCHA Handling:**
- Implement logic to automatically detect CAPTCHA pages on TruePeopleSearch.com.
- Pause the scraping process when a CAPTCHA is detected.
- Provide a clear visual indication to the user that a CAPTCHA needs to be solved.
- Allow the user to resume scraping after solving the CAPTCHA.
- **Automated Scraping:**
- Implement logic to automatically click the "View Details" button on the search results page.
- Implement logic to scrape the data from the contact details page.
- Implement logic to close the tab after scraping the data.
- Implement logic to move to the next record in the CSV file.
- **Error Handling and Logging:**
- Implement comprehensive error handling and logging to facilitate debugging.
- Log all proxy connection attempts, authentication status, and scraping results.
**Technical Details:**
- The extension is written in JavaScript and uses the Chrome Extension API.
- The proxy list is provided in a CSV file.
- The TruePeopleSearch.com website structure will be provided.
- The current extension code will be provided.
**Deliverables:**
- A fully functional Chrome extension that successfully scrapes data from TruePeopleSearch.com while rotating proxies and handling authentication correctly.
- Well-documented code with clear explanations of the implemented solutions.
- Instructions on how to configure and use the extension.
**Skills Required:**
- Chrome Extension Development
- JavaScript
- Proxy Management
- Web Scraping
- Error Handling
- Debugging
**Project Description:**
I have a Chrome extension designed to scrape data from TruePeopleSearch.com. However, I'm encountering two primary issues that need to be resolved:
1. **Proxy Rotation:** The extension is not properly rotating through the provided list of proxies. This is resulting in IP blocks and preventing the scraper from running effectively. I need a developer to implement a robust proxy rotation mechanism that ensures each request is made through a different proxy from the list. The proxy list is provided in a CSV file. The extension currently supports HTTP proxies with username/password authentication.
2. **407 Proxy Authentication Error:** The extension is encountering a "407 Proxy Authentication Required" error, indicating that the proxy authentication is not being handled correctly. I need a developer to troubleshoot and fix the proxy authentication process to ensure the extension can successfully authenticate with the proxies.
3. **CAPTCHA Handling:** The extension needs to automatically detect CAPTCHA pages on TruePeopleSearch.com and pause the scraping process, allowing the user to manually solve the CAPTCHA.
4. **Automated Scraping:** The extension needs to automatically click the "View Details" button on the search results page and scrape the data from the contact details page.
**Specific Requirements:**
- **Proxy Rotation:**
- Implement a reliable proxy rotation mechanism using a list of HTTP proxies from a CSV file.
- Ensure proper handling of username/password authentication for proxies.
- Implement error handling for failed proxy connections and automatic retries with a different proxy.
- **407 Proxy Authentication Error:**
- Identify and resolve the cause of the 407 error.
- Ensure the extension correctly authenticates with the proxy server.
- **CAPTCHA Handling:**
- Implement logic to automatically detect CAPTCHA pages on TruePeopleSearch.com.
- Pause the scraping process when a CAPTCHA is detected.
- Provide a clear visual indication to the user that a CAPTCHA needs to be solved.
- Allow the user to resume scraping after solving the CAPTCHA.
- **Automated Scraping:**
- Implement logic to automatically click the "View Details" button on the search results page.
- Implement logic to scrape the data from the contact details page.
- Implement logic to close the tab after scraping the data.
- Implement logic to move to the next record in the CSV file.
- **Error Handling and Logging:**
- Implement comprehensive error handling and logging to facilitate debugging.
- Log all proxy connection attempts, authentication status, and scraping results.
**Technical Details:**
- The extension is written in JavaScript and uses the Chrome Extension API.
- The proxy list is provided in a CSV file.
- The TruePeopleSearch.com website structure will be provided.
- The current extension code will be provided.
**Deliverables:**
- A fully functional Chrome extension that successfully scrapes data from TruePeopleSearch.com while rotating proxies and handling authentication correctly.
- Well-documented code with clear explanations of the implemented solutions.
- Instructions on how to configure and use the extension.
**Skills Required:**
- Chrome Extension Development
- JavaScript
- Proxy Management
- Web Scraping
- Error Handling
- Debugging
Related categories:
PHP
JavaScript
Python
Mobile App Development
Web Scraping
Software Architecture
Google Chrome
Debugging
Data Scraping
Next.js