Create a pyspark cluster and execute a job
Budget: ₹600 – ₹1,500 INR
Hi,
We have 1000 gz files with 700-1B json objs of 6KB each. Average gz file is 250MB and 2.5GB when unzipped. We have a total of approx 700M-1B files that we have to move to s3 and mongo.
Need to setup a pyspark processing pipeline that will process this and move the data.
We have 1000 gz files with 700-1B json objs of 6KB each. Average gz file is 250MB and 2.5GB when unzipped. We have a total of approx 700M-1B files that we have to move to s3 and mongo.
Need to setup a pyspark processing pipeline that will process this and move the data.