FastAPI Web Scraper for Google SERP with Docker & AWS Deployment
Budget: €100 – €600 EUR
I need to scrape Google based on a specific URL so that we get the exact matching HTML response. So I need a simple web API using FastAPI with a single POST endpoint, e.g., /process-url, which will accept two input arguments in the body (url and api_key) based on a .env variable. That will be a simple verification/login… Developed in Python. In other words, we want to call: http://IP:PORT/process-url with this input:
"url": "https://www.google.com/search?q=implementace&hl=cs&gl=cz&pws=0&peek_pws=0&safe=off&num=70&uule=w+CAIQICIWUHJhZ3VlLCBjemVjaCByZXB1YmxpYw==",
"api_key": "abc1234"
and as the response we want the complete HTML output of the Google SERP response.
The whole thing must run in AWS on EC2, and the delivery should include a working DOCKERFILE, where everything will be run including a virtual display, so it can be tested and also deployed to production.
The web API must handle traffic of at least 1 million requests per day. Ideally, it should run in hypercorn, and it should be ensured that the server’s memory is not exhausted (technically, the whole thing can run on a single instance). We are using unlimited residential proxies.
"url": "https://www.google.com/search?q=implementace&hl=cs&gl=cz&pws=0&peek_pws=0&safe=off&num=70&uule=w+CAIQICIWUHJhZ3VlLCBjemVjaCByZXB1YmxpYw==",
"api_key": "abc1234"
and as the response we want the complete HTML output of the Google SERP response.
The whole thing must run in AWS on EC2, and the delivery should include a working DOCKERFILE, where everything will be run including a virtual display, so it can be tested and also deployed to production.
The web API must handle traffic of at least 1 million requests per day. Ideally, it should run in hypercorn, and it should be ensured that the server’s memory is not exhausted (technically, the whole thing can run on a single instance). We are using unlimited residential proxies.