Real-Time Finance Forecasting -- 2

I’m building a streaming analytics stack that ingests live market information, runs predictive models on the fly, and serves fresh insights through an API. The backbone is already selected—Kafka for ingestion and queuing, Spark Structured Streaming for computation, TimescaleDB for long-horizon storage, FastAPI for the service layer, and MLflow for experiment tracking and model registry.

Here’s the core flow I need implemented:

• Pull real-time data simultaneously from stock market feeds, key economic indicators, and major financial-news streams.
• Push the raw events into Kafka topics, apply Spark jobs to clean and feature-engineer them, then write both the enriched streams and batch aggregates into TimescaleDB.
• Orchestrate model training and scoring with MLflow so that new models can be promoted without downtime. Predictions must be streamed back through Kafka and exposed via low-latency FastAPI endpoints.
• Guarantee end-to-end freshness—new ticks should surface in the API within seconds.

Deliverables I will sign off on:
1. Docker-compose (or Helm) setup that spins up the entire pipeline locally.
2. Source code and notebooks covering data ingestion, Spark jobs, model lifecycle with MLflow, and the FastAPI service exposing REST and WebSocket endpoints.
3. Monitoring dashboards (Grafana/Prometheus or similar) showing lag, throughput, and model performance metrics.
4. A short runbook explaining how to add additional data feeds or redeploy a new model version.

If you have proven experience wiring these exact components together for real-time workloads, I’d love to see a brief note on a comparable project when you bid.