Fix Errors in Python Scraper File -- 2
Budget: $2 – $8 USD
I have a scraper file I need the error fixed. Below is the job description.
Scraping Error for First Script
Unexpected Redirection: After running the code for a while, the website redirects to the homepage. This interrupts the scraping process and causes the script to stop unexpectedly.
Page Limit & City Filter: The website only loads up to 20 pages per search, but there are more listings. Please add a feature that allows the scraper to filter and scrape results by each city individually. The script should:
Select the first city.
Go through all available pages for that city.
Then move to the next city, and repeat until all cities and their pages have been processed.
Database Connection Issue: The database connection drops after scraping just two pages. It seems the connection times out or goes idle, especially while solving CAPTCHAs. Please implement logic to:
Keep the connection alive.
Reconnect automatically if the connection drops.
Ensure data continues to insert even after interruptions.
Incorrect Address Parsing: The address extraction logic sometimes returns irrelevant or incorrect text (e.g., "Court" as a full address). Please improve the logic by:
Validating addresses against the CSV of collected data.
Filtering out results that don’t resemble a full, valid address.
Error in Popstream Scraper Code
The Popstream scraper isn’t working at all. When I run it, the browser just keeps refreshing endlessly.
I will send a video demonstrating the issue for further clarity.
Scraping Error for First Script
Unexpected Redirection: After running the code for a while, the website redirects to the homepage. This interrupts the scraping process and causes the script to stop unexpectedly.
Page Limit & City Filter: The website only loads up to 20 pages per search, but there are more listings. Please add a feature that allows the scraper to filter and scrape results by each city individually. The script should:
Select the first city.
Go through all available pages for that city.
Then move to the next city, and repeat until all cities and their pages have been processed.
Database Connection Issue: The database connection drops after scraping just two pages. It seems the connection times out or goes idle, especially while solving CAPTCHAs. Please implement logic to:
Keep the connection alive.
Reconnect automatically if the connection drops.
Ensure data continues to insert even after interruptions.
Incorrect Address Parsing: The address extraction logic sometimes returns irrelevant or incorrect text (e.g., "Court" as a full address). Please improve the logic by:
Validating addresses against the CSV of collected data.
Filtering out results that don’t resemble a full, valid address.
Error in Popstream Scraper Code
The Popstream scraper isn’t working at all. When I run it, the browser just keeps refreshing endlessly.
I will send a video demonstrating the issue for further clarity.