Create Puppeteer Script That Can Extract Data From DevTools
Budget: $30 – $250 USD
I want to send a post request, like: {"urls":"https://www.domain.com/"} to a Node JS app which uses Puppeteer, and have it return certain info about the URL, such as the URLs of the images which load on the page.
I want to use an Express Server to do it. I already have an app that works like this. I can send this same type of post request to port 4000 and it returns data from it.
Here is what I'm trying to retrieve from the post request.
1. The URLs of the images which load on page load, that means not all images, only the ones that load above the fold, since most are lazy loaded.
2. I need to find a way to find the class of the main DIV for the body of the page.
I think you can get 1. from the network > img. It will have a list of images that load on page load.
I think you can get 2. by using the viewport. If you can find the first DIV below the fold, meaning, below the viewport on page load, then the DIV I want will be its direct parent.
There will be two different post requests, each with their own port number. First post request returns the URLs and image sizes of the images that load on page load in the network > img. Second post request returns the class name of the parent div of the first element below the fold.
I want to use an Express Server to do it. I already have an app that works like this. I can send this same type of post request to port 4000 and it returns data from it.
Here is what I'm trying to retrieve from the post request.
1. The URLs of the images which load on page load, that means not all images, only the ones that load above the fold, since most are lazy loaded.
2. I need to find a way to find the class of the main DIV for the body of the page.
I think you can get 1. from the network > img. It will have a list of images that load on page load.
I think you can get 2. by using the viewport. If you can find the first DIV below the fold, meaning, below the viewport on page load, then the DIV I want will be its direct parent.
There will be two different post requests, each with their own port number. First post request returns the URLs and image sizes of the images that load on page load in the network > img. Second post request returns the class name of the parent div of the first element below the fold.