AWS Video File Download Automation
Budget: £36 – £0 GBP
I'm seeking an AWS expert to develop a system capable of downloading files daily via HTTP. The system should utilize AWS Lambda for its implementation so multiple files can be downloaded at the same time. The files are HLS video streams and comprise .m3u8, .ts and .webvtt file types.
The download urls are stored in an RDS database table, comprising 6m+ rows. We think a script could be written in PHP that submits URL(s) to an SNS queue, which is then connected to Lambda; but we are open to any AWS based arcitecture.
The key objective is to download the 6m+ files as quickly as possible, ideally, within 24 hours. The files do not need to be stored. The purpose of this system is to pre-cache the content on a 3rd-party CDN by making an HTTP request for each file each 24 hours (or more frequently).
Ideal Skills:
- Proficiency in AWS services, particularly Lambda
- Experience with large-scale file downloads
- Strong understanding of HTTP protocols
- Skills in queue management or batch processing
The system should be efficient and reliable, ensuring that all files are downloaded without fail. We will provide access to the RDS database table containing the file URLs. Ideally, the lambda function (if used) will update this table with the time the file was downloaded/cached.
The download urls are stored in an RDS database table, comprising 6m+ rows. We think a script could be written in PHP that submits URL(s) to an SNS queue, which is then connected to Lambda; but we are open to any AWS based arcitecture.
The key objective is to download the 6m+ files as quickly as possible, ideally, within 24 hours. The files do not need to be stored. The purpose of this system is to pre-cache the content on a 3rd-party CDN by making an HTTP request for each file each 24 hours (or more frequently).
Ideal Skills:
- Proficiency in AWS services, particularly Lambda
- Experience with large-scale file downloads
- Strong understanding of HTTP protocols
- Skills in queue management or batch processing
The system should be efficient and reliable, ensuring that all files are downloaded without fail. We will provide access to the RDS database table containing the file URLs. Ideally, the lambda function (if used) will update this table with the time the file was downloaded/cached.