Python Web Scraper Optimization
Budget: €30 – €250 EUR
Project Goal
Enhance an existing Python web scraper to process 20 websites per second, achieving a total of 1,200 websites in under 60 seconds, without data storage.
Scope of Work
Performance Analysis
Analyze the current scraper code for performance bottlenecks, focusing on areas such as network latency, processing delays, and inefficient code structures.
Implementation of Parallel Processing
Integrate asynchronous processing using libraries such as asyncio and aiohttp for non-blocking HTTP requests.
Explore multiprocessing or threading to handle 20 websites in parallel efficiently, ensuring resource utilization is optimized.
Code Optimization
Optimize the scraper code to reduce the processing time per site to 1 second or less, focusing on efficient parsing and minimal overhead.
Ensure proper error handling and retry mechanisms for failed requests to maximize throughput.
Validation and Benchmarking
Test the scraper to ensure it meets the goal of scraping 20 websites per second and a total of 1,200 websites in under 60 seconds.
Provide detailed benchmarks for performance under varying loads.
Additional Information
Current source code repository: https://github.com/wladimir-delcros/websitescraper/
The scraper will not store data; focus on retrieving and processing information in memory.
The implementation should be scalable and easily adaptable for future requirements.
Expected Deliverables
Updated scraper code with asynchronous and parallel processing capabilities.
Documentation detailing changes made, libraries used, and how to run the updated scraper.
Performance benchmarks demonstrating the achievement of the project goal.
Enhance an existing Python web scraper to process 20 websites per second, achieving a total of 1,200 websites in under 60 seconds, without data storage.
Scope of Work
Performance Analysis
Analyze the current scraper code for performance bottlenecks, focusing on areas such as network latency, processing delays, and inefficient code structures.
Implementation of Parallel Processing
Integrate asynchronous processing using libraries such as asyncio and aiohttp for non-blocking HTTP requests.
Explore multiprocessing or threading to handle 20 websites in parallel efficiently, ensuring resource utilization is optimized.
Code Optimization
Optimize the scraper code to reduce the processing time per site to 1 second or less, focusing on efficient parsing and minimal overhead.
Ensure proper error handling and retry mechanisms for failed requests to maximize throughput.
Validation and Benchmarking
Test the scraper to ensure it meets the goal of scraping 20 websites per second and a total of 1,200 websites in under 60 seconds.
Provide detailed benchmarks for performance under varying loads.
Additional Information
Current source code repository: https://github.com/wladimir-delcros/websitescraper/
The scraper will not store data; focus on retrieving and processing information in memory.
The implementation should be scalable and easily adaptable for future requirements.
Expected Deliverables
Updated scraper code with asynchronous and parallel processing capabilities.
Documentation detailing changes made, libraries used, and how to run the updated scraper.
Performance benchmarks demonstrating the achievement of the project goal.