Enhancing Data Infrastructure for Trading Platform

Project Status Report: Data Architecture Evolution
Implemented
1. Infrastructure (Docker)
We have successfully set up the core infrastructure using docker-compose. The following services are running and healthy:

Kafka (Port 9094): For high-throughput market data ingestion.
Redis (Port 6379): For real-time "hot" data storage (latest prices).
ClickHouse (Port 8123): For "warm" data storage (historical OHLC data for charts).
MinIO (Port 9002): S3-compatible object storage for "cold" data archival (raw ticks).
Elasticsearch & Kibana: For system observability and logs.
2. Kafka Producer Service
Created
KafkaProducerService
in NestJS.
Configured to connect to the local Kafka broker.
Ready to publish messages to topics.
In Progress / Next Steps
1. Ingestion Layer (Binance Integration)
Goal: Stream live market data from Binance instead of using simulated data.

Implement Binance WebSocket: Connect to Binance's btcusdt@trade stream.
Publish to Kafka: Forward incoming trades to the market_ticks Kafka topic using the
KafkaProducerService
.
2. Hot Storage (Redis)
Goal: Provide sub-millisecond price updates to the frontend.

Redis Consumer: Create a service to consume from Kafka market_ticks.
Update Cache: Store the latest price in Redis keys (e.g., PRICE:BTCUSDT).
Pub/Sub: Publish updates to a Redis channel for the WebSocket gateway to broadcast to the frontend.
3. Warm Storage (ClickHouse)
Goal: Store historical data for fast charting.

Table Schema: Define ticks and ohlc tables in ClickHouse.
Batch Ingestion: Create a consumer to batch insert Kafka messages into ClickHouse.
4. Cold Storage (S3/MinIO)
Goal: Archive raw data for compliance and backtesting.

Parquet Archiver: Periodically flush data to Parquet files and upload to MinIO.
5. Frontend Integration
Goal: Visualize the live data.

WebSocket Client: Update the React frontend to listen to the new real-time stream.
Chart API: Update
FinancialChart
to fetch history from the new ClickHouse-backed API.
6. Account Management (New)
Goal: Enforce account limits and expiry.

Demo Account: Limit to 1 per user, valid for 15 days.
Purchased Accounts: Limit to 2 per user.
7. Aurora Credits (Gamification)
Goal: Incentivize user engagement.

Earning Logic: Implement rules for Login Streak, Trades, Profit, etc.
Spending Logic: Cap credit usage at 40% of challenge fee.
8. Account Packages
Goal: Configure tiered challenges.

Standard: Balanced risk (80% profit split).
Premium: Higher risk/reward (90% profit split).
Pro: 100% payout flagship.
Seeding: Populate database with these package configurations.
9. Educational Video Platform (New)
Goal: Host and stream educational content securely.

Storage/CDN: Use MinIO (local) / S3+CloudFront (prod).
Access Control: Ensure only users with active accounts can view videos (Signed URLs).
Uploads: Admin interface for uploading content.