BigQuery Real-Time Analytics Pipeline

This project centers on building a production-grade, real-time analytics environment in BigQuery. The goal is an end-to-end pipeline that can ingest high-volume CSV and ORC files the moment they land, transform them efficiently, and surface insights with sub-second responsiveness.

What I already have in place
• Raw data lands continuously as CSV and ORC files.
• Access to BigQuery, GCS, Dataproc, and Cloud SQL is provisioned.

What I need from you
– Design and implement the streaming (and, where sensible, batch) ETL/ELT flow into BigQuery, selecting the right mix of native BigQuery features, Dataproc jobs, or other GCP services to keep latency to an absolute minimum.
– Model partitioned / clustered tables that will scale to petabyte-level volumes while remaining cost-efficient.
– Write and tune MySQL-style and BigQuery SQL for complex aggregations, dashboards, and ad-hoc exploration.
– Embed robust monitoring, alerting, and error-handling so the pipeline can be trusted in production.
– Document the architecture and hand over repeatable deployment steps (Terraform, Deployment Manager, or shell scripts—whatever you prefer as long as it’s reproducible).

Acceptance criteria
1. Fresh files are visible in BigQuery within the agreed SLA.
2. Representative analytic queries complete within target timeframes on large data sets.
3. Automated tests or sample notebooks demonstrate the full ingest-to-insight path.
4. Clear documentation lets another engineer reproduce or extend the solution without guesswork.

If petabyte-scale, real-time analytics in BigQuery is where you shine, let’s get this pipeline delivering value fast.