Iceberg Migration & Ingestion POC
Budget: ₹600 – ₹1,500 INR
The goal is to put together a complete, production-ready reference that moves our legacy Hive ecosystem into Apache Iceberg while keeping performance front and centre.
Scope of work
• Author a concise yet comprehensive guide that walks through every major Iceberg concept—catalogs, snapshots, partition evolution, time-travel, schema evolution, compaction, MERGE INTO, etc.—and pair each topic with at least one runnable exercise so the team can practise what they read.
• Build a reusable codebase that translates existing HQL/Hive queries into Iceberg-compatible syntax. Wherever possible, refactor logic so queries can scale smoothly when the data volume grows from 1 TB toward the upper end of our 10 TB footprint.
• Replace our current Sqoop import jobs with PySpark scripts that land data straight into Iceberg tables, taking advantage of Spark 3, DataFrame API, and Iceberg’s write options.
• Deliver a proof of concept that ingests ten representative tables—covering initial full load, daily incremental, and back-fill of historical partitions—while demonstrating snapshot isolation and fast rollback.
• Provide benchmark notes that highlight the scalability gains (partition pruning, vectorised reads, write amplification reduction, etc.) we achieve after migration.
Acceptance criteria
– Exercises compile and run on Spark 3.x with Iceberg ≥ 1.2.
– Conversion utilities handle typical Hive constructs (dynamic partitions, CTAS, ORC/Parquet formats) without manual rewrites.
– PySpark ingestion completes within the target window on a 3-node test cluster at ~2 TB scale and shows linear growth characteristics as data volume increases.
– POC tables support time-travel queries and exhibit consistent results before and after incremental loads.
All source code should be version-controlled (Git) with README instructions and sample datasets so the internal team can reproduce results instantly.
Scope of work
• Author a concise yet comprehensive guide that walks through every major Iceberg concept—catalogs, snapshots, partition evolution, time-travel, schema evolution, compaction, MERGE INTO, etc.—and pair each topic with at least one runnable exercise so the team can practise what they read.
• Build a reusable codebase that translates existing HQL/Hive queries into Iceberg-compatible syntax. Wherever possible, refactor logic so queries can scale smoothly when the data volume grows from 1 TB toward the upper end of our 10 TB footprint.
• Replace our current Sqoop import jobs with PySpark scripts that land data straight into Iceberg tables, taking advantage of Spark 3, DataFrame API, and Iceberg’s write options.
• Deliver a proof of concept that ingests ten representative tables—covering initial full load, daily incremental, and back-fill of historical partitions—while demonstrating snapshot isolation and fast rollback.
• Provide benchmark notes that highlight the scalability gains (partition pruning, vectorised reads, write amplification reduction, etc.) we achieve after migration.
Acceptance criteria
– Exercises compile and run on Spark 3.x with Iceberg ≥ 1.2.
– Conversion utilities handle typical Hive constructs (dynamic partitions, CTAS, ORC/Parquet formats) without manual rewrites.
– PySpark ingestion completes within the target window on a 3-node test cluster at ~2 TB scale and shows linear growth characteristics as data volume increases.
– POC tables support time-travel queries and exhibit consistent results before and after incremental loads.
All source code should be version-controlled (Git) with README instructions and sample datasets so the internal team can reproduce results instantly.
Related categories:
SQL
Data Warehousing
Hive
Spark
ETL
PySpark
Database Management
Performance Tuning