JWT Authenticated API Scraper

Goal

Build a reliable Python scraper that authenticates to a site’s REST API (JWT), discovers valid agencies, traverses their project listings, and downloads bid results—persisting only records that include line-item (item list) details—into Azure Cosmos DB in (nearly) the original JSON shape. I should be able to run it weekly to fetch new/updated results. Prototyped script attached.

Scope & Data Flow
1. Auth & Token Lifecycle
• Login with username/password to obtain JWT.
• Auto-refresh: decode exp; refresh proactively (e.g., 60–120s before) or on 401/403 with one retry.
2. Discover Agencies
• Use the papi-agency-id endpoint to probe agency IDs.
• If response indicates a valid agency, store agency metadata in agencies container.
• Keep a simple registry of discovered agency IDs for future runs.
3. Project Listings per Agency
• For each valid agency, generate/fetch the listing pages (API endpoints, not browser scraping).
• Collect project metadata (IDs, titles, dates, status).
4. Bid Results (Only When Line Items Exist)
• For projects with a bid result, follow the bid result endpoint.
• Only persist results that include “item list” / line items (there is a specific API for line items—use that).
• Skip/mark projects with no line items.
5. Persistence (Cosmos DB)
• Containers (suggested):
• agencies (partitionKey: /agencyId)
• projects (partitionKey: /agencyId)
• bidResults (partitionKey: /agencyId)
• lineItems (partitionKey: /agencyId)
• Upsert by deterministic IDs:
• agencyId, projectId, bidId, and a stable key for line items (e.g., bidId#lineItemIndex).
• Store remote JSON as-is under data field; add a lightweight envelope:
{
"id": "<stable-id>",
"agencyId": "...",
"projectId": "...",
"bidId": "...",
"source": "planetbids",
"fetchedAt": "ISO-8601",
"data": { <original JSON> }
}

Runs & Modes
• Full sync: discover agencies, crawl everything (respect rate limits).
• Incremental sync (weekly default): only agencies/projects changed since last run (use updatedAt if available; else last-seen IDs/timestamps).
• Trigger via:
• CLI flags: --full or --since YYYY-MM-DD.
• Ready for cron/GitHub Actions/Azure Functions Timer (weekly).

Resilience & Etiquette
• Rate limits: automatic backoff + jitter; configurable QPS.
• Retries: exponential backoff on 429/5xx; circuit-break noisy endpoints.
• Idempotency: all writes are upserts; no dupes on re-run.
• Observability:
• Structured logs (JSON) with request IDs (do not log tokens).
• Metrics summary at end: counts by agencies, projects scanned, bid results found, line-item docs written, skips, errors.

Configuration
• .env (no secrets in repo):
• PB_USERNAME, PB_PASSWORD, PB_COMPANY_ID
• COSMOS_URI, COSMOS_KEY, COSMOS_DB
• Tuning: CONCURRENCY, RATE_LIMIT_QPS, REQUEST_TIMEOUT, RETRY_MAX, REFRESH_SKEW_SECONDS
• Local dev uses dotenv; production uses environment variables.

Tech Stack (preferred)
• Python 3.11, httpx (async) or requests (sync—okay if you add concurrency via concurrent.futures)
• Azure Cosmos DB: azure-cosmos SDK
• CLI: typer or argparse
• Logging: structlog or logging with JSON formatter
• Packaging: uv/pip + pyproject.toml
• Optional: Dockerfile for one-command run

Deliverables
• Source code, clean structure & comments.
• One-step run:
• README:
• Config vars, how to auth, run full vs incremental, and add new endpoints.
• Notes on token refresh & rate limits.
• Sample JSON dumps:
• Example documents from each container (agencies, projects, bidResults, lineItems) showing successful inserts.
• Schema notes:
• Key fields used to generate ids, partition keys, and any minimal normalization.

Acceptance Criteria
• Automatic JWT acquisition & refresh (no manual steps).
• Valid agencies discovered and stored; all target endpoints crawled.
• Only bid results with line items are persisted; others skipped without error.
• Re-running does not duplicate records; updates are upserts.
• A weekly incremental run completes within agreed time and captures new results.
• Handles 24h test run with no unhandled exceptions; logs make failures diagnosable.
• Clear run report with counts + error summary.

Nice-to-Haves (not required, call out if included)
• Concurrency with respectful rate limiting.
• Checkpointing so long runs resume.
• Optional raw-response archive container/bucket for auditing.
• Small dashboard script to count new line-item results per week.

What I Expect Back from You
• Your proposed stack outline (sync vs async, libs).
• How you’ll manage token refresh and incremental detection.
• Any assumptions about the specific endpoints (agency probe, listings, bid results, line items).
• Estimated throughput and guardrails for rate limits.

⸻

If anything here is unclear (e.g., exact param names for the line-items endpoint), make sensible assumptions and list them in the README; I can provide examples during kickoff. My expectation is you’ll deliver the entire scraped dataset from all valid agencies, and I can run the script weekly to keep it fresh.