Job Data Scraper Development Using Scrapy

Job ID: 39115796

Budget: ₹12,500 – ₹37,500 INR

I'm in need of an experienced developer proficient in Scrapy to construct a job scraper. This tool will need to efficiently extract data from various platforms including Internshala, Naukri, Indeed, Glassdoor, Govt Jobs (Sarkari Naukri), and LinkedIn.

Key Features:
- Capable of handling dynamic content and pagination.
- Must bypass cloud-based anti-scraping protections, such as Cloudflare and reCAPTCHA, without risking a ban.
- Incorporation of proxy rotation to prevent detection and IP blocking.
- Utilization of techniques like user-agent randomization and CAPTCHA solving for seamless scraping.

Data Handling:
- Output to be structured and directly inserted into a PostgreSQL database.
- Implementation of error handling and logging for issues, such as CAPTCHAs or alterations in site structure.

Data Fields:
- Job title, company, and location.
- Salary details and application URL.
- Job description and requirements.

Ideal Skills:
- Expertise in Scrapy and Python.
- Proficiency in PostgreSQL.
- Experience with web scraping and data extraction.
- Familiarity with proxy rotation and anti-scraping techniques.
- Excellent problem-solving skills for handling unexpected site changes.