Advanced Web Crawler System Development
Budget: $1,500 – $3,000 USD
We are seeking a skilled developer or development team to create a comprehensive web crawler system. This system should scrape, structure, and update information from web pages, handle large projects, and be accessible via a powerful API. The system should offer user-friendly features for both individuals and businesses and include flexible user management for both customers and employees.
Project Requirements
1. Web Crawling and Data Extraction
-Develop a system capable of scraping entire websites and extracting data from each page.
-The system should divide the scraped information into sections to facilitate easy reading and understanding.
-For e-commerce sites, the system should identify and structure the following information:
--Product category (including breadcrumbs for hierarchical structure)
--Title
--Price
--Availability
--Images (specify the main image)
--Review score
--Unique Selling Points (USPs)
--Colors
--Sizes
--Call to Action (CTA) text
--Description
--Specifications
--Other relevant information
-Enable users to specify which pages, products, and categories to manage, and which information to include in exports. Generate XML or CSV links based on user selection.
-Allow users to update specific pieces of collected information across all or selected pages, ensuring data remains current and accurate.
2. Proxy System
-Implement a dedicated IP system (proxy system) to avoid server or firewall blocking.
3. Data Access and Export
-Allow data export in formats such as CSV, XML, etc.
-Ensure data export is compatible with Google Merchant Center and other similar advertising services.
4. User Interface
-Provide an interface that allows users to explore scraped content through a hierarchical view.
-Ensure the user interface clearly and structurally displays the content of each page.
-Provide tools for users to update specific information within the scraped data, either for all pages or selected ones.
5. User Management and Subscription Management
-Support user management with features for registration, login, and profile management.
-Offer flexible subscription management to sell the service to both individuals and businesses.
-Allow business accounts to add team members (new or existing accounts) to their platform.
-Provide robust user management for employees, including roles such as customer service, sales representatives, administrators, project managers, developers, super admins, etc.
6. API System
-Develop a powerful API that allows other applications to use the service.
-Ensure the API is scalable and capable of handling very large projects.
7. AI Integration
-Integrate with OpenAI and other AI platforms to provide an assistant chat feature.
-Allow the AI assistant to read and answer questions about the content on the current page in real-time.
-Enable the AI assistant to suggest better content for specific sections of the page.
-Provide a simple "replace" feature that allows users to easily use the suggested content to update sections on the crawled site.
Technical Requirements
1. Backend
-Build the system with a robust backend architecture that supports high scalability and performance.
-Use a well-known programming language such as Python, Java, or Node.js for backend services.
2. Database
-Use a scalable database solution, such as PostgreSQL or MongoDB, to store scraped data and user information.
3. Proxy and IP Rotation
-Implement IP rotation and use proxy servers to prevent website blocking.
4. Security
-Implement security measures to protect user data and prevent unauthorized access.
-Ensure data transmission occurs over HTTPS for secure communication.
5. API Documentation
-Provide detailed API documentation describing all available endpoints, parameters, and example calls.
Non-functional Requirements
1. Scalability
-Ensure the system can handle a large amount of data and users simultaneously without affecting performance.
2. Availability
-Design the system for high availability with minimal downtime.
3. Performance
-Ensure the system can scrape and process large amounts of data quickly and efficiently.
4. Usability
-Create an intuitive and easy-to-use interface for users with varying technical backgrounds.
Implementation Plan
1. Project Planning
-Conduct requirements gathering and analysis.
-Establish a timeline and milestones.
2. Development Phase
-Develop the backend.
-Design and implement the database.
-Develop the proxy and IP rotation system.
-Develop the API and create documentation.
-Design the user interface and ensure a good user experience.
-Integrate AI platforms and develop assistant chat features.
3. Testing
-Conduct unit testing.
-Conduct integration testing.
-Conduct load testing.
-Conduct security testing.
4. Launch
-Deploy and install the system in production.
-Provide end-user training and documentation.
5. Maintenance
-Provide continuous monitoring and performance optimization.
-Deliver regular updates and bug fixes.
How to Apply
Please provide:
- A detailed proposal outlining your approach to this project.
- Examples of similar projects you have completed.
- An estimated timeline for each phase of the project.
- Your budget estimate for the entire project.
We look forward to working with you!
Project Requirements
1. Web Crawling and Data Extraction
-Develop a system capable of scraping entire websites and extracting data from each page.
-The system should divide the scraped information into sections to facilitate easy reading and understanding.
-For e-commerce sites, the system should identify and structure the following information:
--Product category (including breadcrumbs for hierarchical structure)
--Title
--Price
--Availability
--Images (specify the main image)
--Review score
--Unique Selling Points (USPs)
--Colors
--Sizes
--Call to Action (CTA) text
--Description
--Specifications
--Other relevant information
-Enable users to specify which pages, products, and categories to manage, and which information to include in exports. Generate XML or CSV links based on user selection.
-Allow users to update specific pieces of collected information across all or selected pages, ensuring data remains current and accurate.
2. Proxy System
-Implement a dedicated IP system (proxy system) to avoid server or firewall blocking.
3. Data Access and Export
-Allow data export in formats such as CSV, XML, etc.
-Ensure data export is compatible with Google Merchant Center and other similar advertising services.
4. User Interface
-Provide an interface that allows users to explore scraped content through a hierarchical view.
-Ensure the user interface clearly and structurally displays the content of each page.
-Provide tools for users to update specific information within the scraped data, either for all pages or selected ones.
5. User Management and Subscription Management
-Support user management with features for registration, login, and profile management.
-Offer flexible subscription management to sell the service to both individuals and businesses.
-Allow business accounts to add team members (new or existing accounts) to their platform.
-Provide robust user management for employees, including roles such as customer service, sales representatives, administrators, project managers, developers, super admins, etc.
6. API System
-Develop a powerful API that allows other applications to use the service.
-Ensure the API is scalable and capable of handling very large projects.
7. AI Integration
-Integrate with OpenAI and other AI platforms to provide an assistant chat feature.
-Allow the AI assistant to read and answer questions about the content on the current page in real-time.
-Enable the AI assistant to suggest better content for specific sections of the page.
-Provide a simple "replace" feature that allows users to easily use the suggested content to update sections on the crawled site.
Technical Requirements
1. Backend
-Build the system with a robust backend architecture that supports high scalability and performance.
-Use a well-known programming language such as Python, Java, or Node.js for backend services.
2. Database
-Use a scalable database solution, such as PostgreSQL or MongoDB, to store scraped data and user information.
3. Proxy and IP Rotation
-Implement IP rotation and use proxy servers to prevent website blocking.
4. Security
-Implement security measures to protect user data and prevent unauthorized access.
-Ensure data transmission occurs over HTTPS for secure communication.
5. API Documentation
-Provide detailed API documentation describing all available endpoints, parameters, and example calls.
Non-functional Requirements
1. Scalability
-Ensure the system can handle a large amount of data and users simultaneously without affecting performance.
2. Availability
-Design the system for high availability with minimal downtime.
3. Performance
-Ensure the system can scrape and process large amounts of data quickly and efficiently.
4. Usability
-Create an intuitive and easy-to-use interface for users with varying technical backgrounds.
Implementation Plan
1. Project Planning
-Conduct requirements gathering and analysis.
-Establish a timeline and milestones.
2. Development Phase
-Develop the backend.
-Design and implement the database.
-Develop the proxy and IP rotation system.
-Develop the API and create documentation.
-Design the user interface and ensure a good user experience.
-Integrate AI platforms and develop assistant chat features.
3. Testing
-Conduct unit testing.
-Conduct integration testing.
-Conduct load testing.
-Conduct security testing.
4. Launch
-Deploy and install the system in production.
-Provide end-user training and documentation.
5. Maintenance
-Provide continuous monitoring and performance optimization.
-Deliver regular updates and bug fixes.
How to Apply
Please provide:
- A detailed proposal outlining your approach to this project.
- Examples of similar projects you have completed.
- An estimated timeline for each phase of the project.
- Your budget estimate for the entire project.
We look forward to working with you!