Parallel Azure Scraper for Price Comparison

Senior Azure-Native Retail Data Extraction Engineer

We are seeking an experienced engineer to design and deliver a production-ready, Azure-native data extraction system for a large U.S. grocery retailer. The solution must run entirely inside our Microsoft Azure subscription and reliably collect ~100,000 product records within a 24-hour window.

The target site is delivered through a modern CDN/WAF stack. We will supply a managed pool of rotating proxies, allowing you to focus on session handling, request shaping, concurrency control, and system resilience, rather than proxy sourcing.

⸻

Environment & Technology

Language and tooling are flexible, provided they integrate cleanly with Azure. Acceptable approaches include (but are not limited to):
• Python (Requests, Scrapy, Playwright, Selenium)
• Node.js (Playwright, Puppeteer, clustered workers)
• C# / .NET (headless browser or HTTP-based approaches)

The solution must be suitable for deployment using Azure Functions, Azure Container Apps, or VM Scale Sets. Please make your architectural choices explicit and explain how they support scalability, resiliency, and cost control in Azure.

⸻

Core Responsibilities
• Design a distributed crawler and orchestration layer capable of completing a full retail catalogue crawl (up to ~100k items) within 24 hours.
• Implement controlled parallelism with safe retry, backoff, and resumability.
• Emit one clean JSON file per run, conforming exactly to the schema below.
• Integrate proxy usage, header management, and session continuity so that crawls remain stable under sustained load.
• Log failures with sufficient context (URL, timestamp, error details) to allow replay and diagnosis.
• Provide simple configuration toggles for:
• crawl frequency
• concurrency limits
• output destination (Azure Blob Storage preferred)
• Package the system for one-command deployment into Azure.

⸻

Required Output Schema

Each run must emit a single JSON file containing records with the following fields (field names must match exactly):
- zipcode
- store_name
- store_location
- store_logo
- category
- sub_category
- product_title
- regular_price
- sale_price
- unit_price
- volume_deals_description
- volume_deals_price
- digital_deals_description
- digital_deal_price
- weight
- upc
- image_urls
- url
- crawl_date

Cache & Incremental Refresh Design (Required)

The extractor must be built around a persistent catalog cache to avoid unnecessary re-discovery and reduce load across runs.

Cache principles
• The cache represents the known product universe for the retailer.
• It must persist across runs and live in Azure Blob Storage (or equivalent durable Azure storage).
• The system must support incremental refreshes, not full re-crawls on every execution.

Required cache behavior
• Cold start / first run
• Discover products and populate the catalog cache.
• Subsequent runs
• Use the cache as the source of truth for known products.
• Refresh prices and offers for cached products.
• Detect and ingest new products without reprocessing the entire catalog.
• Cache entries must be keyed by a stable identifier (e.g., product URL or internal product ID), not just by product title.

Checkpointing & resumability
• Crawls must be resumable:
• Partial failures should not require restarting from the beginning.
• Progress (pagination offsets, category state, etc.) should be checkpointed to durable storage.
• The system should be able to pause and resume without data loss or duplication.

Output interaction with cache
• Cache data must not be overwritten blindly.
• Each run should:
• update time-sensitive fields (prices, deals, crawl_date)
• preserve stable fields (UPC, title, URL, image URLs)
• Cache design should support replay, diffing between runs, and future auditing.

Out of scope
• Stateless scrapers that re-fetch everything every run
• In-memory-only caches
• Designs that require manual cleanup between runs

⸻

Deliverables
1. Source code repository with clear structure and documentation
2. Dockerfile and/or Azure Function App package
3. Azure deployment scripts or pipeline definition (IaC preferred: Bicep, ARM, or Terraform)
4. Sample JSON output demonstrating full field coverage
5. A concise README covering:
• environment setup
• proxy configuration
• how to trigger a crawl
• how to monitor progress and failures

⸻

Acceptance Criteria
• A dry-run against ~500 live products completes successfully with ≥95% field population.
• A full catalogue run completes in under 24 hours on a modest Azure footprint (e.g., B-series VM or comparable Function plan).
• No systemic blocking or failure cascades during sustained execution.
• Output strictly conforms to the defined JSON schema.

⸻

Ideal Candidate Profile
• Proven experience building large-scale, resilient web data extraction systems
• Familiarity with CDN/WAF-protected retail platforms
• Strong understanding of Azure-native architectures
• Comfortable reasoning about throughput, stability, and cost tradeoffs
• Able to clearly document and justify architectural decisions

Proposals that clearly explain cache layout, refresh strategy, failure recovery, and Azure deployment will be prioritized.