Optimize High-Recall Indeed Job Scraper (Python, Multithreading)
Budget: ₹2,000 – ₹2,001 INR
Optimize High-Recall Indeed Job Scraper (Python, Multithreading)
Description:
I have a Python script that scrapes job listings from Indeed USA using ZIP codes as input. It’s built around the external library and reads ZIPs from an .xlsx file covering all US states. The scraping logic is solid—it fetches a high volume of jobs posted in the last 24 hours with excellent accuracy.
I’ve tested it for California only:
8,000+ jobs fetched from 163 ZIP codes
Took 6–7 hours to complete
California has 1,200+ ZIPs, and I haven’t even touched other states yet
What I Need:
I’m looking for a Python expert to optimize the script for speed without compromising the current scraping logic. The goal is to make it scalable across all states and ZIP codes.
Optimization Goals:
Implement multithreading or multiprocessing to parallelize ZIP-level and/or page-level requests
Maintain the current adaptive radius and job-type split logic
Support proxy rotation if needed (I dont have proxies now)
Preserve deduplication and error handling
Ideally stream results to CSV shards or a single output file
I will share the code, once we finalize
Description:
I have a Python script that scrapes job listings from Indeed USA using ZIP codes as input. It’s built around the external library and reads ZIPs from an .xlsx file covering all US states. The scraping logic is solid—it fetches a high volume of jobs posted in the last 24 hours with excellent accuracy.
I’ve tested it for California only:
8,000+ jobs fetched from 163 ZIP codes
Took 6–7 hours to complete
California has 1,200+ ZIPs, and I haven’t even touched other states yet
What I Need:
I’m looking for a Python expert to optimize the script for speed without compromising the current scraping logic. The goal is to make it scalable across all states and ZIP codes.
Optimization Goals:
Implement multithreading or multiprocessing to parallelize ZIP-level and/or page-level requests
Maintain the current adaptive radius and job-type split logic
Support proxy rotation if needed (I dont have proxies now)
Preserve deduplication and error handling
Ideally stream results to CSV shards or a single output file
I will share the code, once we finalize