Senior Xano Expert for Data Pipeline Review (+ Recipe) — Gov CSV/API Ingestion, Normalize & Upsert/Dedupe

Seeking a Xano pro to review an existing plan for ingesting structured records from federal/state environmental datasets (e.g., EPA ECHO/FRS + state DEPs), normalize fields, deduplicate across sources, and upsert into canonical tables with a daily change feed. Deliver a concise critique plus a click-by-click Xano build recipe. NDA required.

Scale (MVP): ~100k facilities across multiple sources.
Run window: Nightly job done by 5:00 AM ET.

Deliverables

Audit memo: what to fix/adjust (accuracy, performance, safety).

One diagram (Miro or PNG): source → staging → normalize → cross-source dedupe → upsert → change_log → daily_feed.

Facility Identifier strategy & schema: one canonical facility_id + a mapping table that relates each source’s native ID to the canonical record (e.g., source_system, source_id, composite identity key); include unique constraints, the lookup/upsert flow, and how it supports cross-source dedupe + auditability.

Step-by-step Xano recipe (modern UI):

Keyset paging (no offset).

Cross-source dedupe (composite key + authoritative IDs when available).

Upsert rules (create vs update; unique constraints; idempotency).

Change logging + daily feed roll-up.

Short Loom walkthrough of the deliverables.

Must-have skills

Expert-level Xano: function stacks, DB design, tasks/scheduling.

ETL/ingestion, normalization, entity resolution/dedup, upsert patterns, idempotency.

Nice to have

Apify experience (actors pulling CSV/API).

Diagramming (Miro / draw.io / Mermaid).

Process & privacy

NDA required before sharing docs or samples.

After NDA: redacted plan + small sample dataset; Xano access (read/contrib).

Start: immediately (US/Eastern friendly).

Acceptance criteria (clear + minimal)

Identifier layer present: mapping table with unique index on (source_system, source_id), composite identity defined; demonstrates linking the same facility across ≥2 sources; idempotent re-ingest creates no duplicates.

Keyset paging implemented in the recipe (no offset).

Cross-source dedupe approach clearly defined.

Upsert rules + unique constraints documented.

Daily feed logic defined (created/updated/closed) to meet a 5:00 AM ET run.

Include in application:
purple monkey