Web Crawler & Video Content Detection System
Budget: €50 – €300 EUR
Objective: Develop a highly efficient and robust web crawler system. The primary goal is to take a list of domain names (e.g., xyz.com) as input and output a comprehensive database of all internal URLs found on that domain, with a boolean flag indicating the presence of a primary video player on each page.
Key Requirements:
Comprehensive Crawling:
The crawler must systematically explore the target domain, starting from the root and following all internal links to a configurable depth. The system should employ intelligent crawling strategies to maximize coverage while maintaining efficiency. The output must include all discoverable URLs within the domain scope.
Intelligent Video Detection:
The system must analyze each crawled page to determine if it contains a primary video player. The focus is on the main content video, not ancillary or advertisement videos. The solution should implement a hybrid approach for optimal accuracy and performance. Pattern-based heuristics should be used to identify video players through HTML5 <video> tags, embedded player iframes, script patterns, and video-related attributes. Optional machine learning analysis may be added for ambiguous cases to improve detection confidence.
Anti-Blocking and Robustness:
The system must be designed to operate effectively against modern web defenses. This includes advanced session, cookie, and local storage management; CAPTCHA handling through integrated solving services; human-like behavioral patterns with randomized delays and user-agent rotation; and the ability to bypass common anti-bot protections and security walls. The crawler must continue functioning even when encountering isolated errors or blocking attempts.
Output and Performance:
The final deliverable must be a structured database (SQLite or PostgreSQL) or a CSV file with columns: url (the full page URL), has_video (true/false), and last_crawled (timestamp). Performance optimization is critical and should rely on asynchronous processing, connection pooling, and efficient resource management.
Deliverables:
Complete source code for the crawler and detection system.
Comprehensive setup and configuration documentation.
Final output database or CSV for the provided domains.
Technical Freedom:
The freelancer may choose any technology stack (Python/Scrapy, Node.js, Go) and any implementation approach, as long as it meets the performance and effectiveness requirements. Technical choices should be justified.
Note:
Specific details about the target websites and content types will be discussed privately to ensure the solution is tailored correctly. The proposal must acknowledge this requirement.
Key Requirements:
Comprehensive Crawling:
The crawler must systematically explore the target domain, starting from the root and following all internal links to a configurable depth. The system should employ intelligent crawling strategies to maximize coverage while maintaining efficiency. The output must include all discoverable URLs within the domain scope.
Intelligent Video Detection:
The system must analyze each crawled page to determine if it contains a primary video player. The focus is on the main content video, not ancillary or advertisement videos. The solution should implement a hybrid approach for optimal accuracy and performance. Pattern-based heuristics should be used to identify video players through HTML5 <video> tags, embedded player iframes, script patterns, and video-related attributes. Optional machine learning analysis may be added for ambiguous cases to improve detection confidence.
Anti-Blocking and Robustness:
The system must be designed to operate effectively against modern web defenses. This includes advanced session, cookie, and local storage management; CAPTCHA handling through integrated solving services; human-like behavioral patterns with randomized delays and user-agent rotation; and the ability to bypass common anti-bot protections and security walls. The crawler must continue functioning even when encountering isolated errors or blocking attempts.
Output and Performance:
The final deliverable must be a structured database (SQLite or PostgreSQL) or a CSV file with columns: url (the full page URL), has_video (true/false), and last_crawled (timestamp). Performance optimization is critical and should rely on asynchronous processing, connection pooling, and efficient resource management.
Deliverables:
Complete source code for the crawler and detection system.
Comprehensive setup and configuration documentation.
Final output database or CSV for the provided domains.
Technical Freedom:
The freelancer may choose any technology stack (Python/Scrapy, Node.js, Go) and any implementation approach, as long as it meets the performance and effectiveness requirements. Technical choices should be justified.
Note:
Specific details about the target websites and content types will be discussed privately to ensure the solution is tailored correctly. The proposal must acknowledge this requirement.
Related categories:
Data Processing
Web Scraping
Web Development
Web Crawling
Data Scraping
Data Analysis
Database Management