Job Data Scraper Development Using Scrapy
Budget: ₹12,500 – ₹37,500 INR
I'm in need of an experienced developer proficient in Scrapy to construct a job scraper. This tool will need to efficiently extract data from various platforms including Internshala, Naukri, Indeed, Glassdoor, Govt Jobs (Sarkari Naukri), and LinkedIn.
Key Features:
- Capable of handling dynamic content and pagination.
- Must bypass cloud-based anti-scraping protections, such as Cloudflare and reCAPTCHA, without risking a ban.
- Incorporation of proxy rotation to prevent detection and IP blocking.
- Utilization of techniques like user-agent randomization and CAPTCHA solving for seamless scraping.
Data Handling:
- Output to be structured and directly inserted into a PostgreSQL database.
- Implementation of error handling and logging for issues, such as CAPTCHAs or alterations in site structure.
Data Fields:
- Job title, company, and location.
- Salary details and application URL.
- Job description and requirements.
Ideal Skills:
- Expertise in Scrapy and Python.
- Proficiency in PostgreSQL.
- Experience with web scraping and data extraction.
- Familiarity with proxy rotation and anti-scraping techniques.
- Excellent problem-solving skills for handling unexpected site changes.
Key Features:
- Capable of handling dynamic content and pagination.
- Must bypass cloud-based anti-scraping protections, such as Cloudflare and reCAPTCHA, without risking a ban.
- Incorporation of proxy rotation to prevent detection and IP blocking.
- Utilization of techniques like user-agent randomization and CAPTCHA solving for seamless scraping.
Data Handling:
- Output to be structured and directly inserted into a PostgreSQL database.
- Implementation of error handling and logging for issues, such as CAPTCHAs or alterations in site structure.
Data Fields:
- Job title, company, and location.
- Salary details and application URL.
- Job description and requirements.
Ideal Skills:
- Expertise in Scrapy and Python.
- Proficiency in PostgreSQL.
- Experience with web scraping and data extraction.
- Familiarity with proxy rotation and anti-scraping techniques.
- Excellent problem-solving skills for handling unexpected site changes.
Related categories:
Business, Accounting, Human Resources & Legal
Python
Web Scraping
Data Mining
Elasticsearch