Build an Apache Iceberg rewrite tool for version-1.4.3 (Backend)
Budget: ₹1,500 – ₹12,500 INR
I’m looking for a freelancer with strong experience in Apache Iceberg (specifically version 1.4.3), Java, and metadata handling to help develop a high-performance metadata rewrite tool to support path migration of Iceberg tables (e.g., HDFS to S3). The tool should rewrite metadata.json, manifest lists, data manifests, and delete manifests by replacing a given source path prefix with a target prefix. It must support both HiveCatalog-based tables and FileIO+metadataLocation-based execution (i.e., handling metadata.json directly at the table location). Local Java thread pool-based parallelism is required (no Spark initially), following the “write to temp + atomic rename” pattern for safety. The tool should expose a CLI for configuration (--database, --table, --source-prefix, --target-prefix, etc.) with dry-run support and be capable of handling multiple tables in a database in one run. Parallelism should be supported at the manifest file level and snapshot level using DAG tracking and snapshot dependency resolution (e.g., via a HashMap-based approach). Performance and correctness are key. Reference implementations and inspiration include this repo and this PR. Prior experience contributing to Iceberg or working with Hadoop-compatible file systems (S3A, HDFS) is a strong plus. I’m happy to discuss the snapshot DAG design and parallelism strategy in more detail.
These are the links for reference -
1. https://github.com/apache/iceberg/pull/11555
2. https://github.com/ksmatharoo/IcebergMetadataRewrite
These are the links for reference -
1. https://github.com/apache/iceberg/pull/11555
2. https://github.com/ksmatharoo/IcebergMetadataRewrite
Related categories:
Business, Accounting, Human Resources & Legal
Apache
Apache Hadoop
Big Data
Apache Spark