Automated Web Scraping Expert Needed -- 2

We are seeking an experienced freelancer to develop a fully automated web scraping and data extraction system that runs on a scheduled basis with zero manual operation. The ideal candidate should be skilled in Python or relevant automation tools and have experience handling large-scale data collection efficiently.

Project Scope:

-Develop a fully automated web scraping system that runs on a predefined schedule without manual intervention.
-Extract structured data from multiple websites and store it in a database, CSV, or other formats.
-Implement error handling, logging, and notification systems to ensure reliability.
-Handle dynamic content, CAPTCHA, and anti-scraping measures when necessary.
-Optimize performance for efficiency and scalability.

Requirements:

-Strong programming skills in Python (BeautifulSoup, Scrapy, Selenium, or similar tools).
-Experience with task scheduling and automation (e.g., cron jobs, Airflow, cloud-based scheduling).
-Ability to work around anti-scraping protections while ensuring compliance with ethical standards.
-Knowledge of databases (SQL, NoSQL) for storing extracted data.
-Experience with API integration as an alternative data collection method.
-Strong attention to detail for data accuracy and consistency.

Preferred Skills:

-Experience working with serverless functions, cloud computing, or containerization (AWS Lambda, Google Cloud Functions, Docker, etc.).
-Knowledge of proxy management, rotating user agents, and CAPTCHA-solving techniques.
-Familiarity with real-time data pipelines and automated reporting.

Job Type:

-Freelance / Project-Based
-Remote work with flexible hours

How to Apply:

If you have relevant experience and can build a fully automated system, please submit:

-A brief introduction highlighting your expertise in web scraping automation.
-Links to past projects or sample work.
-Your expected rate and estimated project timeline.

We will contact shortlisted candidates for further discussions.