Multi-Store Scraper Project with Central Server and Distributed Clients
Budget: €250 – €750 EUR
We are seeking a highly experienced freelancer in web scraping, particularly on Amazon, who can overcome technical challenges and obstacles. The project involves developing a distributed scraping system composed of a central server with a database and multiple clients (distributed across the globe) that will connect to the server and perform scraping tasks.
Project Requirements:
Programming Language: The freelancer can choose the preferred programming language and must specify it in the proposal.
Fixed Price and Warranty: We require a fixed price for the project with a warranty period to address any issues in the code. Once the project is completed, the freelancer will be contracted for future updates (e.g., adding stores), with the price of each update depending on its complexity.
Project Structure:
1. Central Server:
The main server will be hosted on a high-spec Linux Ubuntu server to efficiently handle large data processing tasks.
It will be connected to a MySQL database and will be responsible for:
Accepting client connections via sockets.
Assigning scraping tasks to the clients.
Sending updates to the clients when needed.
Sending notifications when a product meets the configured conditions (e.g., price drops).
2. Clients:
We have several small devices (miniPCs) running Linux, which will act as the clients.
Clients will connect to the central server via sockets, receive updates if needed, and perform scraping tasks using their IP address or a proxy configured in their setup file.
Database Structure:
The central database (managed by the server) will have the following structure:
asins: For monitoring Amazon products.
searches: URLs of categories or keywords to be scraped from each store.
clients: Information about clients (name, IP, version, and last connection).
items: Storage for all scraped products.
stores: A list of stores with their configuration (store ID, Telegram chat ID, etc.).
Objective:
The system will perform scraping across Amazon stores in all European countries and various local stores, storing the items and prices in our database. The goal is to receive alerts when price drops are detected based on predefined discount thresholds or criteria.
Project Requirements:
Programming Language: The freelancer can choose the preferred programming language and must specify it in the proposal.
Fixed Price and Warranty: We require a fixed price for the project with a warranty period to address any issues in the code. Once the project is completed, the freelancer will be contracted for future updates (e.g., adding stores), with the price of each update depending on its complexity.
Project Structure:
1. Central Server:
The main server will be hosted on a high-spec Linux Ubuntu server to efficiently handle large data processing tasks.
It will be connected to a MySQL database and will be responsible for:
Accepting client connections via sockets.
Assigning scraping tasks to the clients.
Sending updates to the clients when needed.
Sending notifications when a product meets the configured conditions (e.g., price drops).
2. Clients:
We have several small devices (miniPCs) running Linux, which will act as the clients.
Clients will connect to the central server via sockets, receive updates if needed, and perform scraping tasks using their IP address or a proxy configured in their setup file.
Database Structure:
The central database (managed by the server) will have the following structure:
asins: For monitoring Amazon products.
searches: URLs of categories or keywords to be scraped from each store.
clients: Information about clients (name, IP, version, and last connection).
items: Storage for all scraped products.
stores: A list of stores with their configuration (store ID, Telegram chat ID, etc.).
Objective:
The system will perform scraping across Amazon stores in all European countries and various local stores, storing the items and prices in our database. The goal is to receive alerts when price drops are detected based on predefined discount thresholds or criteria.