Help with AWS CSV-to-S3-to-Data Validation-to-Report-in-S3
Budget: $25 – $50 USD
Create proof of concept using AWS to automate the following process:
(1) CSVs are uploaded into an S3 bucket: Files could be hundreds per day with thousands of rows each. Each file triggers Crawler.
(2) Crawler (with Ruleset) automatically creates a table and validates all data.
Here it gets fuzzy for me. Goal is for a report of what exact data (specifying column, row, data) failed and to put that report in a different S3 bucket.
If Lambda code is needed to manage any of this, I am a Python dev and have already written some Lambda scripts to do part of the above process with script but none of it is working and I put that aside because project goal is to get proof of concept working with as little (or no) code as possible.
IF we do use any Lambda functions, we must run it async because validation of any given file will take far longer than we want a Lambda script to be running.
I propose our method be me screen sharing with you over Zoom or Teams and you guiding my hand.
(1) CSVs are uploaded into an S3 bucket: Files could be hundreds per day with thousands of rows each. Each file triggers Crawler.
(2) Crawler (with Ruleset) automatically creates a table and validates all data.
Here it gets fuzzy for me. Goal is for a report of what exact data (specifying column, row, data) failed and to put that report in a different S3 bucket.
If Lambda code is needed to manage any of this, I am a Python dev and have already written some Lambda scripts to do part of the above process with script but none of it is working and I put that aside because project goal is to get proof of concept working with as little (or no) code as possible.
IF we do use any Lambda functions, we must run it async because validation of any given file will take far longer than we want a Lambda script to be running.
I propose our method be me screen sharing with you over Zoom or Teams and you guiding my hand.