Apache Iceberg Metadata Rewrite Tool

Job ID: 39532012

Budget: ₹12,500 – ₹37,500 INR

I'm looking for an experienced Java developer to build a CLI-based tool under the Apache Iceberg ecosystem to rewrite metadata paths between filesystems, like from HDFS to S3.

Key Requirements:
- Rewrite metadata.json, manifests, and data file paths
- Support both HiveCatalog and FileIO-based execution
- Implement local parallelism using ThreadPool (up to 20K manifests)
- Enable snapshot-level DAG-based parallelism with hash validation
- Provide AUTO vs SPARK execution mode
- Use atomic "write then rename" approach (no staging directories)
- Develop a validation test suite (CTAS, ALTER, compaction)

Ideal Skills:
- Proficiency in Java, especially in concurrency
- Familiarity with Apache Iceberg
- Experience with DAG-based workflows
- Optional: Knowledge of Spark, YARN, or Kubernetes

For reference, check this design: https://github.com/ksmatharoo/IcebergMetadataRewrite