Stock photography website scraper
Budget: $30 – $250 USD
I need a robust and efficient python scraper for a stock photography website. The website is a React infinite scroll, and it is possible to access the data directly via the JSON object.
The code needs to:
1. Use proxies to cycle through a list of queries that segment images into searches with smaller results. I have written the code to generate these query strings. This will probably require getting/managing session tokens/cookies to avoid detection. I will share testing proxy info with the freelancer.
2. capture the file name, metadata, and URLs for each image in a pandas dataframe.
3. Go to each image page, get the keywords and store in dataframe. It may be possible to get the keywords via JSON; I haven't tested this.
4. save each image at highest resolution possible, 1500px.
5. output dataframe to CSV file for testing (I will pipe this dataframe into my SQL database, but it is fine for you to save to CSV.)
6. restart gracefully: Because I need to scrape many million images, this code will need to be able to restart where it left off, if the process is interrupted by a crash, website outage, or internet connection issue.
7. be reusable: There are more stock photography sites I want to scrape so make sure to write the code so it is reusable for these future efforts. This is really important. And also, there is more work to be done, and if you do this work well, I would likely be offering the future work to you.
If you would like the site info, query structure, JSON response sample, and supporting code that I have already written, please message me and ask for the "supporting materials" and I will send it to you.
The code needs to:
1. Use proxies to cycle through a list of queries that segment images into searches with smaller results. I have written the code to generate these query strings. This will probably require getting/managing session tokens/cookies to avoid detection. I will share testing proxy info with the freelancer.
2. capture the file name, metadata, and URLs for each image in a pandas dataframe.
3. Go to each image page, get the keywords and store in dataframe. It may be possible to get the keywords via JSON; I haven't tested this.
4. save each image at highest resolution possible, 1500px.
5. output dataframe to CSV file for testing (I will pipe this dataframe into my SQL database, but it is fine for you to save to CSV.)
6. restart gracefully: Because I need to scrape many million images, this code will need to be able to restart where it left off, if the process is interrupted by a crash, website outage, or internet connection issue.
7. be reusable: There are more stock photography sites I want to scrape so make sure to write the code so it is reusable for these future efforts. This is really important. And also, there is more work to be done, and if you do this work well, I would likely be offering the future work to you.
If you would like the site info, query structure, JSON response sample, and supporting code that I have already written, please message me and ask for the "supporting materials" and I will send it to you.