Reconciliation of two flat files using Java and Apache Spark
Budget: $30 – $250 USD
Hello, My requirement is described below
1) Read two flat files(.dat files with ^ delimiter) on Hadoop Edge Node into Spark data frames
2) Drop unwanted columns from those data frames
3) Generate a md5 hash key for each record in the data frames after dropping columns
4) Join two data frames on hash key and identify mismatched records and write those records to a file on Hadoop Edge node
5) This code should be in Java and Apache Spark
1) Read two flat files(.dat files with ^ delimiter) on Hadoop Edge Node into Spark data frames
2) Drop unwanted columns from those data frames
3) Generate a md5 hash key for each record in the data frames after dropping columns
4) Join two data frames on hash key and identify mismatched records and write those records to a file on Hadoop Edge node
5) This code should be in Java and Apache Spark