Product Details Extraction Script
Budget: $30 – $250 USD
I need a script that navigates through a specific e-commerce site, clicks through each product entry, drills into its detail page, and programmatically captures four data points: product description, key feature list, customer rating score, and the seller or store address shown on the page.
Because the environment the tool will run in is headless, I do not want a browser-driven approach such as Selenium or Puppeteer. Please work with direct HTTP requests, an official/undocumented API, or another non-browser technique that still mimics the required clicks so the full detail page is returned.
Deliver the data straight into a relational database of your choice (MySQL or PostgreSQL preferred) and include:
• A clean, well-commented source file or package that performs the crawl and insert
• A SQL script that creates the target table(s) with appropriate data types and indexes
• A short README explaining configuration variables (URL, DB credentials, crawl depth, etc.) and how to launch the job on Linux
Accuracy matters more than speed; I am not interested in collecting user reviews or stock prices at this stage. The script must handle pagination, follow product links reliably, and respect reasonable delay intervals so the site is not overloaded.
Once the crawler finishes, I will test by querying the database and cross-checking a random sample of items against the live site. A job is considered complete when every sampled record shows the correct description, features list, rating, and address.
Because the environment the tool will run in is headless, I do not want a browser-driven approach such as Selenium or Puppeteer. Please work with direct HTTP requests, an official/undocumented API, or another non-browser technique that still mimics the required clicks so the full detail page is returned.
Deliver the data straight into a relational database of your choice (MySQL or PostgreSQL preferred) and include:
• A clean, well-commented source file or package that performs the crawl and insert
• A SQL script that creates the target table(s) with appropriate data types and indexes
• A short README explaining configuration variables (URL, DB credentials, crawl depth, etc.) and how to launch the job on Linux
Accuracy matters more than speed; I am not interested in collecting user reviews or stock prices at this stage. The script must handle pagination, follow product links reliably, and respect reasonable delay intervals so the site is not overloaded.
Once the crawler finishes, I will test by querying the database and cross-checking a random sample of items against the live site. A job is considered complete when every sampled record shows the correct description, features list, rating, and address.