Real-Time Novel Data Scraping Script (similar to what sites like novelfire.net uses)
Budget: $30 – $60 USD
I'm looking for a developer to create a comprehensive script capable of fetching novels and related data from specified websites, starting with novelsonline.org, with the ability to add more sources in the future. The script must scrape all content and update the data in real-time, 24/7 to provide latest updates. The functionality should be similar to websites like novelfire.net.
THE PRICE IS FOR THE SCRIPT ONLY, MAY BE OPEN FOR ENTIRE PROJECT IF REQUIRED AT INCREASED PRICE
Key Requirements:
- Develop a script to scrape and crawl specified websites for novel-related data.
- Fetch all content, including novel text, author information, and publication dates.
- Ensure the script updates already fetched data in real-time, continuously.
- Store the data in a PostgreSQL database and integrate it with payload on my part.
- Provide a scalable solution that can accommodate additional data sources in the future.
Ideal Skills and Experience:
- Proficiency in web scraping and crawling technologies.
- Experience with PostgreSQL for database management.
- Familiarity with scalable data fetching solutions.
- Ability to implement real-time data updating mechanisms.
- Knowledge of integrating data with external payload systems.
Please include in your bid:
- The tech stack you propose to use (Crawl4ai is an option, but suggest alternatives if more robust).
- Explanation of how your solution is scalable and reliable.
- Detailed description of how the script will function and what it will achieve.
If I am wrong, on how the site like novelfire works, please correct me and specify it in your bid as how it works and how will you make it.
THE PRICE IS FOR THE SCRIPT ONLY, MAY BE OPEN FOR ENTIRE PROJECT IF REQUIRED AT INCREASED PRICE
Key Requirements:
- Develop a script to scrape and crawl specified websites for novel-related data.
- Fetch all content, including novel text, author information, and publication dates.
- Ensure the script updates already fetched data in real-time, continuously.
- Store the data in a PostgreSQL database and integrate it with payload on my part.
- Provide a scalable solution that can accommodate additional data sources in the future.
Ideal Skills and Experience:
- Proficiency in web scraping and crawling technologies.
- Experience with PostgreSQL for database management.
- Familiarity with scalable data fetching solutions.
- Ability to implement real-time data updating mechanisms.
- Knowledge of integrating data with external payload systems.
Please include in your bid:
- The tech stack you propose to use (Crawl4ai is an option, but suggest alternatives if more robust).
- Explanation of how your solution is scalable and reliable.
- Detailed description of how the script will function and what it will achieve.
If I am wrong, on how the site like novelfire works, please correct me and specify it in your bid as how it works and how will you make it.
Related categories:
Python
Web Scraping
PostgreSQL
Web Development
Data Integration
Automation
Database Management
API Integration