Python Developer for Binance Futures ML Pipeline
Budget: ₹600 – ₹1,500 INR
We are looking for a Python developer with experience to build a robust, local pipeline that processes Binance Futures historical data into an ML-ready dataset.
The goal is to ingest public data from Binance Vision (aggTrades, all klines, and bookDepth) and output clean, normalized, lookahead-bias-free features stored in Parquet format or DuckDB.
Scope of Work & Deliverables
1. Ingestion & Database Setup (Core Foundation)
Data Source: Programmatic downloading of historical daily/monthly ZIP files from public data.binance.vision (specifically aggTrades, all klines [1m], and bookDepth for BTCUSDT, ETHUSDT, SOLUSDT, XRPUSDT, BNBUSDT).
Storage Architecture: Set up a local storage solution using DuckDB or Parquet to handle millions of rows without memory issues.
Alignment: Parse and align different frequencies (tick-by-tick trades, order book snapshots, and 1m klines) to a unified timestamp sequence.
2. Core Microstructure Feature Extraction
Implement Python/Polars (or Pandas) scripts to compute the features on the aligned data.
3. Advanced Optimization & ML Readiness
Strict Lookahead Bias Prevention: Ensure all rolling features (e.g., rolling z-scores, Parkinson volatility) are calculated using t−1 parameters to prevent data leakage.
Normalization: Implement rolling z-scores or min-max normalization per symbol to keep features stationary.
Labeling: Implement a basic Triple Barrier Method or directional label generator.
Output: Save clean Parquet files per symbol, free of NaNs and infinite values, structured for immediate model training.
The goal is to ingest public data from Binance Vision (aggTrades, all klines, and bookDepth) and output clean, normalized, lookahead-bias-free features stored in Parquet format or DuckDB.
Scope of Work & Deliverables
1. Ingestion & Database Setup (Core Foundation)
Data Source: Programmatic downloading of historical daily/monthly ZIP files from public data.binance.vision (specifically aggTrades, all klines [1m], and bookDepth for BTCUSDT, ETHUSDT, SOLUSDT, XRPUSDT, BNBUSDT).
Storage Architecture: Set up a local storage solution using DuckDB or Parquet to handle millions of rows without memory issues.
Alignment: Parse and align different frequencies (tick-by-tick trades, order book snapshots, and 1m klines) to a unified timestamp sequence.
2. Core Microstructure Feature Extraction
Implement Python/Polars (or Pandas) scripts to compute the features on the aligned data.
3. Advanced Optimization & ML Readiness
Strict Lookahead Bias Prevention: Ensure all rolling features (e.g., rolling z-scores, Parkinson volatility) are calculated using t−1 parameters to prevent data leakage.
Normalization: Implement rolling z-scores or min-max normalization per symbol to keep features stationary.
Labeling: Implement a basic Triple Barrier Method or directional label generator.
Output: Save clean Parquet files per symbol, free of NaNs and infinite values, structured for immediate model training.