Selenium Python Linux Scraping
Budget: $30 – $250 USD
I’m looking to have a python app written that will be able to scrape certain available job openings on linkedin, depending on criteria in the job description, as well as ignoring other jobs containing certain job descriptions, titles, key words, and locations. The application will need to be throttled such that it only requests a particular number of jobs, and at a random duration between following links.
The script should ONLY follow pages that match certain criteria in the title and location. Also, scripts should not follow duplicate job titles (same company, same post date, same company)
An example of search criteria:
“Linux” jobs in the past 24 hours , full-time , “onsite/hybrid”, located in 50 miles from zip code 10012.
Location must NOT be in “New Jersey” , “Connecticut” , or “Albany, NY”
Title must contain “project” or “manager” or “cloud” or “azure”
Title must NOT contain “cybersecurity” or “security” or “scientist” or “devops”
ONLY job links that meet the criteria and are not duplicates should be followed and others ignored. This will help reduce the number of jobs viewed.
Once a matching job is found, the following information is to be scraped.
- Scrape date
- Job post date
- Job title
- Company name
- Company LinkedIn link
- Job link on LinkedIn
- Location of job
- Onsite, hybrid, remote
- Fulltime (or whatever is shown for that)
- # of employees and remainder of info on that line
- Application link
- Body of job description
Job details should be stored in plain text.
This scraped data should be stored in a postgresql database. The database design can be very simple. All scraped data can be put into a single table and not need to be normalized. All search criteria data should be in the database as well.
For testing, I will need a script to dump each table into a CSV
NOTE: you will not have access to the machine that I will be testing this from. This project will be from scratch, so we can design the database schema to fit. I will need complete directions to set this up so that our environments match. (Will simplify the installation).
The environment will be the latest stable ubuntu and postgres. Whenever possible, Please use packages that are natively installed via the package manager.
Contact me if you need additional information and I will update this spec.
When bidding on this project, please tell me what kind of ice cream that you like, because I would like to know that this description was actually read prior to bidding.
The script should ONLY follow pages that match certain criteria in the title and location. Also, scripts should not follow duplicate job titles (same company, same post date, same company)
An example of search criteria:
“Linux” jobs in the past 24 hours , full-time , “onsite/hybrid”, located in 50 miles from zip code 10012.
Location must NOT be in “New Jersey” , “Connecticut” , or “Albany, NY”
Title must contain “project” or “manager” or “cloud” or “azure”
Title must NOT contain “cybersecurity” or “security” or “scientist” or “devops”
ONLY job links that meet the criteria and are not duplicates should be followed and others ignored. This will help reduce the number of jobs viewed.
Once a matching job is found, the following information is to be scraped.
- Scrape date
- Job post date
- Job title
- Company name
- Company LinkedIn link
- Job link on LinkedIn
- Location of job
- Onsite, hybrid, remote
- Fulltime (or whatever is shown for that)
- # of employees and remainder of info on that line
- Application link
- Body of job description
Job details should be stored in plain text.
This scraped data should be stored in a postgresql database. The database design can be very simple. All scraped data can be put into a single table and not need to be normalized. All search criteria data should be in the database as well.
For testing, I will need a script to dump each table into a CSV
NOTE: you will not have access to the machine that I will be testing this from. This project will be from scratch, so we can design the database schema to fit. I will need complete directions to set this up so that our environments match. (Will simplify the installation).
The environment will be the latest stable ubuntu and postgres. Whenever possible, Please use packages that are natively installed via the package manager.
Contact me if you need additional information and I will update this spec.
When bidding on this project, please tell me what kind of ice cream that you like, because I would like to know that this description was actually read prior to bidding.