AWS Lambda and Glue
Budget: $30 – $250 USD
1. A lambda function that triggers a glue job whenever a new gzip file is uploaded to S3
2. A glue job script that unzips the file and extract the json
3. If the extracted json is >1GB, split it into multiple files and make sure header is attached to all the splits with updated record count
4. Upload all the files back to another folder in S3.
2. A glue job script that unzips the file and extract the json
3. If the extracted json is >1GB, split it into multiple files and make sure header is attached to all the splits with updated record count
4. Upload all the files back to another folder in S3.