AWS CLI Process to upload files to S3 if they do not already exist or have been modified
Budget: $30 – $250 USD
Hello. I have millions of small files consisting of about 2TB of data. I have pushed all these files to an S3 bucket.
I would like to run a process each week to upload files if they don't already exist or are newer than the version on s3.
I obviously can't run a CP script which will upload files whether or not it exists. The cost of puts is pretty expensive!
Sync in the CLI command with this many files is pretty timely and resource intensive especially on folders than contain a large number of files.
Ideally would like:
1. Something that can be schedules (.bat file, etc).
2. Something that can run in a .bat script, however if a utility or more advanced programming makes sense I would like to go that way. Whichever way get's me the best / fastest results each week.
I have several other folders/buckets I am considering doing this same thing with.
I would like to run a process each week to upload files if they don't already exist or are newer than the version on s3.
I obviously can't run a CP script which will upload files whether or not it exists. The cost of puts is pretty expensive!
Sync in the CLI command with this many files is pretty timely and resource intensive especially on folders than contain a large number of files.
Ideally would like:
1. Something that can be schedules (.bat file, etc).
2. Something that can run in a .bat script, however if a utility or more advanced programming makes sense I would like to go that way. Whichever way get's me the best / fastest results each week.
I have several other folders/buckets I am considering doing this same thing with.