Apache Iceberg Metadata Rewrite Tool
Budget: ₹12,500 – ₹37,500 INR
I'm looking for an experienced Java developer to build a CLI-based tool under the Apache Iceberg ecosystem to rewrite metadata paths between filesystems, like from HDFS to S3.
Key Requirements:
- Rewrite metadata.json, manifests, and data file paths
- Support both HiveCatalog and FileIO-based execution
- Implement local parallelism using ThreadPool (up to 20K manifests)
- Enable snapshot-level DAG-based parallelism with hash validation
- Provide AUTO vs SPARK execution mode
- Use atomic "write then rename" approach (no staging directories)
- Develop a validation test suite (CTAS, ALTER, compaction)
Ideal Skills:
- Proficiency in Java, especially in concurrency
- Familiarity with Apache Iceberg
- Experience with DAG-based workflows
- Optional: Knowledge of Spark, YARN, or Kubernetes
For reference, check this design: https://github.com/ksmatharoo/IcebergMetadataRewrite
Key Requirements:
- Rewrite metadata.json, manifests, and data file paths
- Support both HiveCatalog and FileIO-based execution
- Implement local parallelism using ThreadPool (up to 20K manifests)
- Enable snapshot-level DAG-based parallelism with hash validation
- Provide AUTO vs SPARK execution mode
- Use atomic "write then rename" approach (no staging directories)
- Develop a validation test suite (CTAS, ALTER, compaction)
Ideal Skills:
- Proficiency in Java, especially in concurrency
- Familiarity with Apache Iceberg
- Experience with DAG-based workflows
- Optional: Knowledge of Spark, YARN, or Kubernetes
For reference, check this design: https://github.com/ksmatharoo/IcebergMetadataRewrite
Related categories:
Business, Accounting, Human Resources & Legal
Python
Linux
Software Architecture
RESTful