Job Data Miner and Aggregator
Budget: $250 – $750 USD
This project involves the development of a web scraper to extract job listings from multiple job portals. The primary objective is to extract key information such as Job Title, URL, Location, Posted date, Job Type (Full Time or Contract to Contract (C2C)), and Rate/Pay, for each listing. The web scraping will be done using Puppeteer, a headless/headed browser automation tool (depending on what job portals allow), and data read/write operations will be managed through a Python API.
Detailed Project Description:
1. Build the URL for each website depending on the technology and filters given: The project will start with constructing specific URLs for each job portal. This involves parameterizing the URLs to include given filters such as technology, job type, and location. For example, a URL could look like this: `https://www.jobportal.com/search?technology=Java&location=NewYork`. This requires a good understanding of how each job portal's search functionality works, which can be discovered by analyzing the structure of URLs after performing a manual search. When a new portal given it should be easy to build the url’s.
2. Scrap the URLs and get the details of each job: Once the URLs are ready, the Puppeteer tool will automate the browser and navigate to these URLs. Then, it will extract the job details. The selection of the required fields (Job Title, URL, Location, Posted date, Job Type, Rate/Pay) can be achieved by inspecting the HTML structure of the webpage and identifying the relevant HTML tags or attributes that hold the required data.
3. Scrap the job details from the Job portals in the screenshot provided:
4. Database R/W through Python API: The extracted job details will then be sent to a Python-based API responsible for communicating with the database. The data can be formatted in a structured format like JSON for easy parsing and manipulation. The Python API will include functionality to write (insert) the job data into the database and read (retrieve) data as needed. The choice of the database system (SQL or NoSQL) depends on the project requirements.
Detailed Project Description:
1. Build the URL for each website depending on the technology and filters given: The project will start with constructing specific URLs for each job portal. This involves parameterizing the URLs to include given filters such as technology, job type, and location. For example, a URL could look like this: `https://www.jobportal.com/search?technology=Java&location=NewYork`. This requires a good understanding of how each job portal's search functionality works, which can be discovered by analyzing the structure of URLs after performing a manual search. When a new portal given it should be easy to build the url’s.
2. Scrap the URLs and get the details of each job: Once the URLs are ready, the Puppeteer tool will automate the browser and navigate to these URLs. Then, it will extract the job details. The selection of the required fields (Job Title, URL, Location, Posted date, Job Type, Rate/Pay) can be achieved by inspecting the HTML structure of the webpage and identifying the relevant HTML tags or attributes that hold the required data.
3. Scrap the job details from the Job portals in the screenshot provided:
4. Database R/W through Python API: The extracted job details will then be sent to a Python-based API responsible for communicating with the database. The data can be formatted in a structured format like JSON for easy parsing and manipulation. The Python API will include functionality to write (insert) the job data into the database and read (retrieve) data as needed. The choice of the database system (SQL or NoSQL) depends on the project requirements.
Related categories:
PHP
Business, Accounting, Human Resources & Legal
Python
Web Scraping
Data Mining