Python or JS Developer Needed for Adzuna Job Scraper Improvement (Fetch All Job Data, JSON/HTML Parsing
Budget: ₹1,000 – ₹1,001 INR
I am looking for a skilled Python or JS developer with expertise in web scraping, Selenium, and HTML parsing to enhance and complete a job scraping tool for the Adzuna job portal.
Project Goals:
code must fetch all job listings from Adzuna for a given search query URL (e.g., https://www.adzuna.com/search?q=Microsoft+Fabric&w=US).
Handle cases where JSON-LD data is missing by falling back to HTML parsing to extract all relevant details.
Current Issues:
my code scrapes some job data (titles, descriptions, company names, etc.), but many fields remain blank.
Not all job listings are fetched; the script currently stops at ~157 jobs out of ~410 listed.
Fallback HTML parsing is insufficient, leaving many fields incomplete.
Requirements:
Comprehensive Data Extraction:
Fetch all job listings from Adzuna for the provided URL.
Extract key fields, including:
Job Title
Company Name
Description (cleaned and free of HTML tags)
Location
Posting Date (
Ensure no blank fields
Pagination Support:
Ensure the scraper navigates through all pages of the search results, capturing every job listing.
Fallback Mechanism:
If JSON-LD is unavailable or malformed, extract data reliably using HTML selectors for titles, company names, descriptions, etc.
Error Handling:
Implement robust error handling for issues like missing elements, timeouts, and unexpected HTML structures.
Log skipped or problematic jobs for review.
Output:
Save the data to a CSV file with all columns accurately populated.
Optimization:
Improve the scraper's speed and memory efficiency, ensuring compatibility with the existing Selenium + Chrome WebDriver setup.
Deliverables:
A Python script that fetches all job listings from Adzuna for the given URL.
Accurate and complete CSV output containing all fetched job data.
Detailed comments in the code for clarity.
Skills Required:
Python or JS
Web Scraping (Selenium, BeautifulSoup, etc.)
JSON-LD and HTML Parsing
CSV Data Handling
Debugging and Error Handling
Project Goals:
code must fetch all job listings from Adzuna for a given search query URL (e.g., https://www.adzuna.com/search?q=Microsoft+Fabric&w=US).
Handle cases where JSON-LD data is missing by falling back to HTML parsing to extract all relevant details.
Current Issues:
my code scrapes some job data (titles, descriptions, company names, etc.), but many fields remain blank.
Not all job listings are fetched; the script currently stops at ~157 jobs out of ~410 listed.
Fallback HTML parsing is insufficient, leaving many fields incomplete.
Requirements:
Comprehensive Data Extraction:
Fetch all job listings from Adzuna for the provided URL.
Extract key fields, including:
Job Title
Company Name
Description (cleaned and free of HTML tags)
Location
Posting Date (
Ensure no blank fields
Pagination Support:
Ensure the scraper navigates through all pages of the search results, capturing every job listing.
Fallback Mechanism:
If JSON-LD is unavailable or malformed, extract data reliably using HTML selectors for titles, company names, descriptions, etc.
Error Handling:
Implement robust error handling for issues like missing elements, timeouts, and unexpected HTML structures.
Log skipped or problematic jobs for review.
Output:
Save the data to a CSV file with all columns accurately populated.
Optimization:
Improve the scraper's speed and memory efficiency, ensuring compatibility with the existing Selenium + Chrome WebDriver setup.
Deliverables:
A Python script that fetches all job listings from Adzuna for the given URL.
Accurate and complete CSV output containing all fetched job data.
Detailed comments in the code for clarity.
Skills Required:
Python or JS
Web Scraping (Selenium, BeautifulSoup, etc.)
JSON-LD and HTML Parsing
CSV Data Handling
Debugging and Error Handling