Python Script: Multisite E-commerce Data Scraper
Budget: $250 – $750 USD
Service Request: Python Scraper Development for Marketplaces
Project Overview
We are looking for a freelance developer with experience in web scraping automation using Python to create a script capable of extracting product information from six Chilean marketplaces: Lider, Ripley, Jumbo, MercadoLibre, Falabella, and Paris. The goal is to collect key product data based on keyword searches on each site and consolidate the results for analysis.
The freelancer must deliver an executable Python script for a local Windows environment that meets all the specified requirements, along with user-friendly documentation for non-technical users, explaining configuration, manual execution, and scheduled automation of the script. A robust, modular, and low-maintenance solution following industry best practices is highly valued.
Scope and Minimum Requirements
Below are the mandatory minimum requirements that the project must fulfill:
1. Keyword-Based Product Search:
The script will take an array (list) of keywords as input. For each keyword, it must perform a product search on each of the six marketplaces mentioned, navigating the first two pages of results for each keyword on each site.
2. Complete Data Extraction:
For each product found in the search results, the script must extract all available information, including at minimum:
Product name or title
Price (current price, and if on sale, the original price and/or discount percentage, if visible)
Product link (URL)
Product image URL (cover image)
Product rating (e.g., 4.5 out of 5, if shown)
Number of stars (e.g., 4 stars, if graphically displayed)
Number of reviews (total customer reviews, if available)
Product position on the page (e.g., 1st, 2nd, 3rd, etc.)
Page number where the product was found (1 or 2)
Keyword used in the search
Date and time of data extraction
Marketplace name (identify which of the six sites the data came from)
3. Dataframes per Marketplace and Consolidated Dataset:
Extracted data will be organized into six pandas DataFrames, one for each marketplace, and an additional consolidated DataFrame aggregating all results into a single unified table. Each DataFrame must include columns for all the mentioned fields (name, price, link, etc.). DataFrames must be correctly indexed and structured for easy filtering and analysis. (Optionally, exporting these DataFrames to CSV or Excel could be included for user convenience).
4. Detailed Logging System:
The script must implement a robust and detailed logging system that reports scraping progress in both the console and log files, logging key events such as:
Start and completion of the entire process.
Start and completion of extraction for each marketplace.
Start and completion of the search for each keyword in each marketplace.
Number of products found per page/keyword.
Warnings if a page has no results or if expected data is missing.
Errors or exceptions, including detailed information to diagnose the issue.
Any connection retry attempts or temporary delays due to site restrictions.
Logs must be informative and granular, allowing a user or developer to understand at which step a failure might have occurred. For example:
"Searching for 'Laptop gamer' on MercadoLibre, page 1... 20 results obtained."
5. Comprehensive Error Handling:
If errors occur during execution (e.g., connection failure, HTML structure changes, missing elements, timeout issues), the script must not stop but should:
Capture the exception and log it with maximum detail (error message, exception type, which site/page/keyword it happened on, possibly even the stack trace in the log file).
Handle the error in a way that allows the scraping process to continue to the next page, product, or site where possible.
If an entire marketplace fails (e.g., if the site is down or access is blocked), continue with the other sites while marking that one as failed.
Provide clear error messages to non-technical users in the console, e.g.:
"Error extracting data from Paris, page 2 for 'Laptop gamer': Timeout while loading the page." but without stopping the entire program.
Include automated retry attempts for transient failures (e.g., wait a few seconds and retry once if an HTTP request fails).
6. Functional Documentation for Non-Technical Users:
The final deliverable must include clear and step-by-step documentation for a non-technical audience, explaining:
Script workflow: what the script does and how it is structured (in a general sense, without diving into every line of code).
Necessary pre-configurations: e.g., installing Python, installing dependencies (pip install -r requirements.txt), setting up variables (such as where to define keywords).
Windows execution instructions: from downloading the repository/files to running the script (e.g., by double-clicking a prepared .bat file or running python scraper.py in CMD/Powershell). Screenshots may be included for clarity.
How to automate execution: guide on scheduling the script using Windows Task Scheduler for automatic execution at a specific time daily.
Troubleshooting common issues: e.g., what to do if a site changes and the script stops working, how to update the keyword list, how to interpret logs, etc.
The documentation can be delivered as a README.md file in the project repository or as a separate PDF/Word document but must be clear, well-structured (numbered steps, section headers), and written in accessible language.
7. Code Delivery and Quality Standards:
The Python source code must be delivered in one or multiple .py scripts, ready for execution in a local environment with the necessary libraries. We expect:
A modular project structure with well-organized folders (e.g., separating scrapers by marketplace or by functionality: extraction, parsing, storage).
Clean and readable code following best practices (PEP8 formatting, clear variable names, consistent indentation, well-structured functions).
Commented code and docstrings in relevant places to describe functions, expected parameters, and any complex logic.
A requirements.txt file listing all dependencies to ensure easy environment setup.
Version control with Git (if possible, delivered as a private repository with a clear commit history).
Basic testing or validation steps to verify that data extraction is working correctly.
Project Timeline and Milestones
We estimate that the development will take approximately 2 to 3 weeks (about 10-15 business days of actual work), divided into the following milestones:
Initial Analysis & Design (Days 1-2):
Marketplace structure review, scraping strategy definition.
Project skeleton setup (logging, folder structure).
Basic Scraper for One Marketplace (Days 3-5):
Working scraper for a single marketplace (e.g., MercadoLibre).
Data extraction for one keyword, covering all fields, logging basic progress.
Implementation for All 6 Marketplaces (Days 6-10):
Fully functional scrapers for all six marketplaces.
Consolidation into six DataFrames plus a unified dataset.
Error Handling, Logging, and Optimization (Days 8-12, parallel to step 3):
Improved logging, exception handling, script reliability.
Ensuring smooth execution and correct output formatting.
Final Documentation and Delivery (Days 13-14):
Full user documentation, script cleanup, and final testing.
A test run in the client’s Windows environment (if possible).
How to Apply
If you're interested in this project and have the required skills, please send your proposal including:
A brief summary of your relevant experience, especially in scraping and Python.
Your approach to the project (which tools you would use, any ideas to handle potential challenges).
A realistic timeline and milestone breakdown (can be based on our suggested milestones or modified).
Your pricing (preferably per milestone).
Any questions or clarifications about the project scope.
We look forward to working with you! Submit your proposal today! ?
Project Overview
We are looking for a freelance developer with experience in web scraping automation using Python to create a script capable of extracting product information from six Chilean marketplaces: Lider, Ripley, Jumbo, MercadoLibre, Falabella, and Paris. The goal is to collect key product data based on keyword searches on each site and consolidate the results for analysis.
The freelancer must deliver an executable Python script for a local Windows environment that meets all the specified requirements, along with user-friendly documentation for non-technical users, explaining configuration, manual execution, and scheduled automation of the script. A robust, modular, and low-maintenance solution following industry best practices is highly valued.
Scope and Minimum Requirements
Below are the mandatory minimum requirements that the project must fulfill:
1. Keyword-Based Product Search:
The script will take an array (list) of keywords as input. For each keyword, it must perform a product search on each of the six marketplaces mentioned, navigating the first two pages of results for each keyword on each site.
2. Complete Data Extraction:
For each product found in the search results, the script must extract all available information, including at minimum:
Product name or title
Price (current price, and if on sale, the original price and/or discount percentage, if visible)
Product link (URL)
Product image URL (cover image)
Product rating (e.g., 4.5 out of 5, if shown)
Number of stars (e.g., 4 stars, if graphically displayed)
Number of reviews (total customer reviews, if available)
Product position on the page (e.g., 1st, 2nd, 3rd, etc.)
Page number where the product was found (1 or 2)
Keyword used in the search
Date and time of data extraction
Marketplace name (identify which of the six sites the data came from)
3. Dataframes per Marketplace and Consolidated Dataset:
Extracted data will be organized into six pandas DataFrames, one for each marketplace, and an additional consolidated DataFrame aggregating all results into a single unified table. Each DataFrame must include columns for all the mentioned fields (name, price, link, etc.). DataFrames must be correctly indexed and structured for easy filtering and analysis. (Optionally, exporting these DataFrames to CSV or Excel could be included for user convenience).
4. Detailed Logging System:
The script must implement a robust and detailed logging system that reports scraping progress in both the console and log files, logging key events such as:
Start and completion of the entire process.
Start and completion of extraction for each marketplace.
Start and completion of the search for each keyword in each marketplace.
Number of products found per page/keyword.
Warnings if a page has no results or if expected data is missing.
Errors or exceptions, including detailed information to diagnose the issue.
Any connection retry attempts or temporary delays due to site restrictions.
Logs must be informative and granular, allowing a user or developer to understand at which step a failure might have occurred. For example:
"Searching for 'Laptop gamer' on MercadoLibre, page 1... 20 results obtained."
5. Comprehensive Error Handling:
If errors occur during execution (e.g., connection failure, HTML structure changes, missing elements, timeout issues), the script must not stop but should:
Capture the exception and log it with maximum detail (error message, exception type, which site/page/keyword it happened on, possibly even the stack trace in the log file).
Handle the error in a way that allows the scraping process to continue to the next page, product, or site where possible.
If an entire marketplace fails (e.g., if the site is down or access is blocked), continue with the other sites while marking that one as failed.
Provide clear error messages to non-technical users in the console, e.g.:
"Error extracting data from Paris, page 2 for 'Laptop gamer': Timeout while loading the page." but without stopping the entire program.
Include automated retry attempts for transient failures (e.g., wait a few seconds and retry once if an HTTP request fails).
6. Functional Documentation for Non-Technical Users:
The final deliverable must include clear and step-by-step documentation for a non-technical audience, explaining:
Script workflow: what the script does and how it is structured (in a general sense, without diving into every line of code).
Necessary pre-configurations: e.g., installing Python, installing dependencies (pip install -r requirements.txt), setting up variables (such as where to define keywords).
Windows execution instructions: from downloading the repository/files to running the script (e.g., by double-clicking a prepared .bat file or running python scraper.py in CMD/Powershell). Screenshots may be included for clarity.
How to automate execution: guide on scheduling the script using Windows Task Scheduler for automatic execution at a specific time daily.
Troubleshooting common issues: e.g., what to do if a site changes and the script stops working, how to update the keyword list, how to interpret logs, etc.
The documentation can be delivered as a README.md file in the project repository or as a separate PDF/Word document but must be clear, well-structured (numbered steps, section headers), and written in accessible language.
7. Code Delivery and Quality Standards:
The Python source code must be delivered in one or multiple .py scripts, ready for execution in a local environment with the necessary libraries. We expect:
A modular project structure with well-organized folders (e.g., separating scrapers by marketplace or by functionality: extraction, parsing, storage).
Clean and readable code following best practices (PEP8 formatting, clear variable names, consistent indentation, well-structured functions).
Commented code and docstrings in relevant places to describe functions, expected parameters, and any complex logic.
A requirements.txt file listing all dependencies to ensure easy environment setup.
Version control with Git (if possible, delivered as a private repository with a clear commit history).
Basic testing or validation steps to verify that data extraction is working correctly.
Project Timeline and Milestones
We estimate that the development will take approximately 2 to 3 weeks (about 10-15 business days of actual work), divided into the following milestones:
Initial Analysis & Design (Days 1-2):
Marketplace structure review, scraping strategy definition.
Project skeleton setup (logging, folder structure).
Basic Scraper for One Marketplace (Days 3-5):
Working scraper for a single marketplace (e.g., MercadoLibre).
Data extraction for one keyword, covering all fields, logging basic progress.
Implementation for All 6 Marketplaces (Days 6-10):
Fully functional scrapers for all six marketplaces.
Consolidation into six DataFrames plus a unified dataset.
Error Handling, Logging, and Optimization (Days 8-12, parallel to step 3):
Improved logging, exception handling, script reliability.
Ensuring smooth execution and correct output formatting.
Final Documentation and Delivery (Days 13-14):
Full user documentation, script cleanup, and final testing.
A test run in the client’s Windows environment (if possible).
How to Apply
If you're interested in this project and have the required skills, please send your proposal including:
A brief summary of your relevant experience, especially in scraping and Python.
Your approach to the project (which tools you would use, any ideas to handle potential challenges).
A realistic timeline and milestone breakdown (can be based on our suggested milestones or modified).
Your pricing (preferably per milestone).
Any questions or clarifications about the project scope.
We look forward to working with you! Submit your proposal today! ?