Download 1 million images from a list of image URLs, upload to Linode Object Storage
Budget: $30 – $250 USD
I have a list of approximately 800,000 image URLs for images which I need to have downloaded from an external website and then uploaded to a Linode Object Storage bucket. Each image is approximately 100 KB in size.
Each of the images will have to be downloaded/uploaded twice, with different resolutions. The URL allows you to specify the resolution.
There is a mapping for the file names that you will have to follow. For example, one of the image URLs provided may be:
"https://i.website.com/13784475/r/il_640xN.1247390554_5leo.jpg"
From here, you will download the image in two different resolutions, located at:
"https://i.website.com/13784475/red/il_640xN.1247390554_5leo.jpg"
"https://i.website.com/13784475/red/il_320xN.1247390554_5leo.jpg"
These two downloaded images will then be uploaded to the Object Storage bucket with concatenated filenames:
"13784475red1247390554_5leo_640.jpg"
"13784475red1247390554_5leo_320.jpg"
You will be making many requests to the website server. Some of these requests may result in errors.
If you encounter a 429 error code from the server, it is your responsibility to find a way around it, either through adjusting the rate at which you download the files or by using proxies.
If you encounter an error that is not a 429 response code:
* Create an "errors" document for the list of image URLs which produced errors, and add the imageURL to this document. (It can be either text or CSV file).
* Upload a default placeholder image in its place.
Deliverable:
* The approximately 1.6 million images in Object Storage bucket
* Document with list of images which could not be retrieved
Notes:
* I can provide the list of URLs to you in whichever format you desire (CSV, text file, etc.).
* I will provide you with access credentials to the Linode Object Storage bucket to which you will upload the images.
* I will provide two milestone payments for the project: I will release the first milestone once half the work is completed, and the second milestone once all the work is completed.
When you apply for this project:
* Tell me about a project similar to this that you have completed in the past
* Tell me exactly how you intend to complete the project. What technology will you use?
Each of the images will have to be downloaded/uploaded twice, with different resolutions. The URL allows you to specify the resolution.
There is a mapping for the file names that you will have to follow. For example, one of the image URLs provided may be:
"https://i.website.com/13784475/r/il_640xN.1247390554_5leo.jpg"
From here, you will download the image in two different resolutions, located at:
"https://i.website.com/13784475/red/il_640xN.1247390554_5leo.jpg"
"https://i.website.com/13784475/red/il_320xN.1247390554_5leo.jpg"
These two downloaded images will then be uploaded to the Object Storage bucket with concatenated filenames:
"13784475red1247390554_5leo_640.jpg"
"13784475red1247390554_5leo_320.jpg"
You will be making many requests to the website server. Some of these requests may result in errors.
If you encounter a 429 error code from the server, it is your responsibility to find a way around it, either through adjusting the rate at which you download the files or by using proxies.
If you encounter an error that is not a 429 response code:
* Create an "errors" document for the list of image URLs which produced errors, and add the imageURL to this document. (It can be either text or CSV file).
* Upload a default placeholder image in its place.
Deliverable:
* The approximately 1.6 million images in Object Storage bucket
* Document with list of images which could not be retrieved
Notes:
* I can provide the list of URLs to you in whichever format you desire (CSV, text file, etc.).
* I will provide you with access credentials to the Linode Object Storage bucket to which you will upload the images.
* I will provide two milestone payments for the project: I will release the first milestone once half the work is completed, and the second milestone once all the work is completed.
When you apply for this project:
* Tell me about a project similar to this that you have completed in the past
* Tell me exactly how you intend to complete the project. What technology will you use?