Digital Product Scraper (≤ $300)
Budget: $250 – $750 USD
WE WILL NOT NEGOTIATE WITH ANYONE FOR MORE THAN $300
I require a robust, reliable, and automated scraping solution to extract a comprehensive inventory of approximately 250,000 automotive parts from ulti.cl. The primary objective is to generate a clean, well-structured dataset of the entire catalog—which may fluctuate in size—for analytical purposes.
The Core Challenge
The site structure does not allow for a simple menu-based category crawl. Many products are not properly categorized or indexed within the navigation menus, and relying on category trees will result in missing a large portion of the inventory. Therefore, the extraction strategy must bypass traditional navigation and rely on search-query simulation and site-wide index traversal. You will need to implement techniques such as:
Search Simulation: Automating the internal search engine by cycling through relevant automotive keywords, part numbers, or character combinations to surface all 250k+ items.
High-Volume Management: Given the scale, the architecture must support persistent storage (e.g., SQLite or PostgreSQL) to handle the volume efficiently without memory bottlenecks.
Dynamic Anti-Bot Handling: Implementing rotating residential proxies, user-agent randomization, and request throttling to sustain high-speed extraction across such a large dataset without being blocked.
Scope of Data
Text Data: Product name, complete technical specifications/descriptions, current price, and stock status for each of the ~250,000 products.
Images: Primary product images saved locally, ensuring high-quality, watermark-free versions.
Metadata: SKU, part compatibility, and source URL.
Technical Stack & Requirements
Preferred Stack: Python (Playwright or Scrapy with Playwright integration) is preferred for handling dynamic JavaScript content and complex DOM interactions at scale.
Output Format: Structured data exported to CSV or JSON (or direct database dump), with images organized in a mirrored folder structure.
Documentation: A concise README detailing how to update search parameters and selectors when the site layout changes.
Reporting: An automated runtime log summarizing total pages scanned, items successfully saved, and any skipped/failed URLs.
Acceptance Criteria
Completeness: Must capture at least 95% of the total product catalog (~250k items), specifically targeting "hidden" or non-categorized items.
Accuracy: Zero tolerance for watermarked images; the scraper must retrieve the original source assets.
Autonomous Operation: The script must run without manual intervention once configured, handling credential/proxy rotation automatically.
Resilience: Must handle dynamic content loading and potential rate-limiting effectively across large-scale sessions.
I am looking for a developer who understands how to navigate complex e-commerce architectures where search-depth is the primary bottleneck. Please outline your approach for bypassing the category-based limitations and how you plan to manage the memory and storage requirements for a dataset of this size.
Sitemap/Crawler Traversal: Identifying and parsing sitemaps or dynamically traversing product IDs to ensure 100% coverage, as standard category crawling will miss a significant portion of the catalog.
Dynamic Anti-Bot Handling: Implementing rotating residential proxies, user-agent randomization, and request throttling to avoid detection while maintaining high-speed extraction.
Scope of Data
Text Data: Product name, complete technical specifications/descriptions, current price, and stock status.
Images: Primary product images saved locally, ensuring high-quality, watermark-free versions.
Metadata: SKU, part compatibility (if available), and source URL.
Technical Stack & Requirements
Preferred Stack: Python (Playwright or Scrapy with Playwright integration) is preferred for handling dynamic JavaScript content and complex DOM interactions.
Output Format: Clean CSV or JSON, with images organized in a mirrored folder structure.
Documentation: A concise README detailing how to update search parameters and selectors when the site layout changes.
Reporting: An automated runtime log summarizing total pages scanned, items successfully saved, and any skipped/failed URLs.
Acceptance Criteria
Completeness: Must capture at least 95% of the total product catalog, ensuring that "hidden" or non-categorized items are included.
Accuracy: Zero tolerance for watermarked images; the scraper must retrieve the original source assets.
Autonomous Operation: The script must run without manual intervention once configured, handling credential/proxy rotation automatically.
Resilience: Must handle dynamic content loading and potential rate-limiting effectively.
I am looking for a developer who understands how to navigate complex e-commerce architectures where search-depth is the primary bottleneck. Please outline your approach for bypassing the category-based limitations and how you plan to simulate the search-index traversal.
Follow-up question: To better tailor the technical implementation, do you have a list of core automotive categories or specific part-number prefixes that you would like prioritized during the initial search-traversal phase?
I require a robust, reliable, and automated scraping solution to extract a comprehensive inventory of approximately 250,000 automotive parts from ulti.cl. The primary objective is to generate a clean, well-structured dataset of the entire catalog—which may fluctuate in size—for analytical purposes.
The Core Challenge
The site structure does not allow for a simple menu-based category crawl. Many products are not properly categorized or indexed within the navigation menus, and relying on category trees will result in missing a large portion of the inventory. Therefore, the extraction strategy must bypass traditional navigation and rely on search-query simulation and site-wide index traversal. You will need to implement techniques such as:
Search Simulation: Automating the internal search engine by cycling through relevant automotive keywords, part numbers, or character combinations to surface all 250k+ items.
High-Volume Management: Given the scale, the architecture must support persistent storage (e.g., SQLite or PostgreSQL) to handle the volume efficiently without memory bottlenecks.
Dynamic Anti-Bot Handling: Implementing rotating residential proxies, user-agent randomization, and request throttling to sustain high-speed extraction across such a large dataset without being blocked.
Scope of Data
Text Data: Product name, complete technical specifications/descriptions, current price, and stock status for each of the ~250,000 products.
Images: Primary product images saved locally, ensuring high-quality, watermark-free versions.
Metadata: SKU, part compatibility, and source URL.
Technical Stack & Requirements
Preferred Stack: Python (Playwright or Scrapy with Playwright integration) is preferred for handling dynamic JavaScript content and complex DOM interactions at scale.
Output Format: Structured data exported to CSV or JSON (or direct database dump), with images organized in a mirrored folder structure.
Documentation: A concise README detailing how to update search parameters and selectors when the site layout changes.
Reporting: An automated runtime log summarizing total pages scanned, items successfully saved, and any skipped/failed URLs.
Acceptance Criteria
Completeness: Must capture at least 95% of the total product catalog (~250k items), specifically targeting "hidden" or non-categorized items.
Accuracy: Zero tolerance for watermarked images; the scraper must retrieve the original source assets.
Autonomous Operation: The script must run without manual intervention once configured, handling credential/proxy rotation automatically.
Resilience: Must handle dynamic content loading and potential rate-limiting effectively across large-scale sessions.
I am looking for a developer who understands how to navigate complex e-commerce architectures where search-depth is the primary bottleneck. Please outline your approach for bypassing the category-based limitations and how you plan to manage the memory and storage requirements for a dataset of this size.
Sitemap/Crawler Traversal: Identifying and parsing sitemaps or dynamically traversing product IDs to ensure 100% coverage, as standard category crawling will miss a significant portion of the catalog.
Dynamic Anti-Bot Handling: Implementing rotating residential proxies, user-agent randomization, and request throttling to avoid detection while maintaining high-speed extraction.
Scope of Data
Text Data: Product name, complete technical specifications/descriptions, current price, and stock status.
Images: Primary product images saved locally, ensuring high-quality, watermark-free versions.
Metadata: SKU, part compatibility (if available), and source URL.
Technical Stack & Requirements
Preferred Stack: Python (Playwright or Scrapy with Playwright integration) is preferred for handling dynamic JavaScript content and complex DOM interactions.
Output Format: Clean CSV or JSON, with images organized in a mirrored folder structure.
Documentation: A concise README detailing how to update search parameters and selectors when the site layout changes.
Reporting: An automated runtime log summarizing total pages scanned, items successfully saved, and any skipped/failed URLs.
Acceptance Criteria
Completeness: Must capture at least 95% of the total product catalog, ensuring that "hidden" or non-categorized items are included.
Accuracy: Zero tolerance for watermarked images; the scraper must retrieve the original source assets.
Autonomous Operation: The script must run without manual intervention once configured, handling credential/proxy rotation automatically.
Resilience: Must handle dynamic content loading and potential rate-limiting effectively.
I am looking for a developer who understands how to navigate complex e-commerce architectures where search-depth is the primary bottleneck. Please outline your approach for bypassing the category-based limitations and how you plan to simulate the search-index traversal.
Follow-up question: To better tailor the technical implementation, do you have a list of core automotive categories or specific part-number prefixes that you would like prioritized during the initial search-traversal phase?
Related categories:
PHP
Python
Web Scraping
Software Architecture
Scrapy
Data Extraction
MongoDB
Data Analysis
Automation
Database Management