Scraping publicly available Business Pages and extracting data
Budget: €200 – €500 EUR
Disclaimer: For the selected freelancer it is not allowed to scrape any user data and behind login. All the required data can also be found via Google, so it is public information about businesses which in every country of the world is legal to scrape. As it is required to extract only a part of the data from a business page, it is totally legal to scrape and use this data. I don’t want to have any user data, so please do not scrape any reviews, just the aggregated scores.
I need data from as much business pages as possible from a known social network. I need only the following data extracted from there:
- Business Page URL like: xxxxxxxx.com/TALLYWEiJL.official/
- Amount of Followers
- Reviews amount
- Rating
- Phone number
- Website
- Contact Email
- Category
- Insta Profile URL
- WhatsApp number
- Business Manager ID (should be in the source code)
- Country as two letter code (like DE for Germany)
- City
- Post Code
- Shops: yes/no (on a fraction of pages there is a shopping tab. I need this info)
I only need pages with reviews. A google search finds 8,6 Mio business pages with reviews: shorturl.at/cnq34
We can reduce the amount of scraped pages further as I only need business pages with more than 10 reviews.
I need someone who has done such a job before. Please tell me a bit more how you would do the job. As it would be about 9 mio pages to check if there is a whatsapp button on them, this raises a few questions:
- How do you get the 9 mio URLs? Some you can probably get via Google. You will need a proxie to overcome Google’s bot detection. I can provide it. You could use Google to extract URLs to business pages which have more than 10 reviews.
- It is clear that you need a robust setup plus proxy service to handle such an amount of requests plus you need time. If you really want to check 9 mio pages about the amount of reviews directly with the speed of 100 pages per minute, it would need 60 days to scrape. So I either recommend to build the system so 1000 queries can go parallel or you go a more effective way.
- My guess is, that maybe 1 Million pages have a WhatsApp button, so I am OK only these to be scraped and the respective data to be extracted. But you need to find a way to get them. Here is an example for such a page: shorturl.at/nT569
- Additionally I also need the pages with a Shop tab like this one: shorturl.at/qrV89. I guess it will be another 100.000 pages.
If you have questions I am happy to answer all of them.
Please only bid, if you have read everything and understood the project. If you have done similar jobs, please provide samples.
I need data from as much business pages as possible from a known social network. I need only the following data extracted from there:
- Business Page URL like: xxxxxxxx.com/TALLYWEiJL.official/
- Amount of Followers
- Reviews amount
- Rating
- Phone number
- Website
- Contact Email
- Category
- Insta Profile URL
- WhatsApp number
- Business Manager ID (should be in the source code)
- Country as two letter code (like DE for Germany)
- City
- Post Code
- Shops: yes/no (on a fraction of pages there is a shopping tab. I need this info)
I only need pages with reviews. A google search finds 8,6 Mio business pages with reviews: shorturl.at/cnq34
We can reduce the amount of scraped pages further as I only need business pages with more than 10 reviews.
I need someone who has done such a job before. Please tell me a bit more how you would do the job. As it would be about 9 mio pages to check if there is a whatsapp button on them, this raises a few questions:
- How do you get the 9 mio URLs? Some you can probably get via Google. You will need a proxie to overcome Google’s bot detection. I can provide it. You could use Google to extract URLs to business pages which have more than 10 reviews.
- It is clear that you need a robust setup plus proxy service to handle such an amount of requests plus you need time. If you really want to check 9 mio pages about the amount of reviews directly with the speed of 100 pages per minute, it would need 60 days to scrape. So I either recommend to build the system so 1000 queries can go parallel or you go a more effective way.
- My guess is, that maybe 1 Million pages have a WhatsApp button, so I am OK only these to be scraped and the respective data to be extracted. But you need to find a way to get them. Here is an example for such a page: shorturl.at/nT569
- Additionally I also need the pages with a Shop tab like this one: shorturl.at/qrV89. I guess it will be another 100.000 pages.
If you have questions I am happy to answer all of them.
Please only bid, if you have read everything and understood the project. If you have done similar jobs, please provide samples.