Webscraper
Budget: $30 – $250 AUD
We are looking for a skilled web scraper to gather text data from multiple specific websites. The scraped data should be organized in csv, json, and sqlite file formats.
Ideal Skills and Experience:
- Proficiency in web scraping tools and techniques
- Familiarity with Python or other programming languages commonly used for web scraping
- Experience in gathering text data from websites
- Ability to organize and export data in csv, json, and sqlite formats
Here is all the information required to decide if this is a job for you and to submit a proper quote.
Here are the details of what I require
I would like a GUI based webscaper/crawler using python, scrapy and playwright. But if there is a big difference in price with implementing a GUI I am comfortable using command line tools as well.
I need to search these websites.
1. yellowpages.com (US)
2. yellowpages.com.au (Australia)
3. yellowpages.ca (Canada)
4. yellow.co.nz (New Zealand)
5. yell.com (UK)
6. google.com
I need to be able to search for businesses, I then need to be able to open the business website and get some more data.
I need to be able to select output name and file format i.e. csv, json, sqlite etc.
My search criteria would be industry, location, and how many pages. I need the option to change the sort feature on each of the websites. I also need to know whether the data scraped is an advertisement or feature placement within the page of the search websites above with some kind of indicator.
I would only be searching one of the above websites at a time, so would need a way to select the website I want to search.
The data fields I require for each business is;
1. Business name
2. URL
3. Physical address
4. Phone Number
5. All Email addresses (this is the data I require as the most important)
Currently I am doing this manually and believe it can be automated.
Please make sure you understand my requirements. This is how I do this process manually
1. I type my search criteria into one of the above websites with the location (usually a city, a state or province/county).
2. On the results page I change the sort order to what is best for the industry I am searching.
3. I open all the results from the search results in a new tab for each result.
4. I open each tab and get as much information from that tab which is never enough of the information I require.
5. I then open the website for that business and get the rest of the information that was not in the search tab for one of the search sites.
This is where the automation stop, the rest of the work I do manually.
Start your response with a '$' sign so I know you have actually read the criteria and understand what I am looking for. Please be accurate with your pricing as I want a realistic price to start with.
This is how I do it now.
Throughout this process I manually cut and paste the data I need into a csv file so the company I do work for can check the information. The 3 parts they check is the email addresses, the phone number and the physical address to ensure they are correct and exist.
They send me back the checked data so I know who is worth my time contacting.
My goal is to automate that entire process so I have the completed file to send to the company I do work for and I then can just work on contacting the businesses that check out.
I then take that csv file and convert it into a json file or an sqlite file and do my work from that.
This is why I require different file formats.
Ideal Skills and Experience:
- Proficiency in web scraping tools and techniques
- Familiarity with Python or other programming languages commonly used for web scraping
- Experience in gathering text data from websites
- Ability to organize and export data in csv, json, and sqlite formats
Here is all the information required to decide if this is a job for you and to submit a proper quote.
Here are the details of what I require
I would like a GUI based webscaper/crawler using python, scrapy and playwright. But if there is a big difference in price with implementing a GUI I am comfortable using command line tools as well.
I need to search these websites.
1. yellowpages.com (US)
2. yellowpages.com.au (Australia)
3. yellowpages.ca (Canada)
4. yellow.co.nz (New Zealand)
5. yell.com (UK)
6. google.com
I need to be able to search for businesses, I then need to be able to open the business website and get some more data.
I need to be able to select output name and file format i.e. csv, json, sqlite etc.
My search criteria would be industry, location, and how many pages. I need the option to change the sort feature on each of the websites. I also need to know whether the data scraped is an advertisement or feature placement within the page of the search websites above with some kind of indicator.
I would only be searching one of the above websites at a time, so would need a way to select the website I want to search.
The data fields I require for each business is;
1. Business name
2. URL
3. Physical address
4. Phone Number
5. All Email addresses (this is the data I require as the most important)
Currently I am doing this manually and believe it can be automated.
Please make sure you understand my requirements. This is how I do this process manually
1. I type my search criteria into one of the above websites with the location (usually a city, a state or province/county).
2. On the results page I change the sort order to what is best for the industry I am searching.
3. I open all the results from the search results in a new tab for each result.
4. I open each tab and get as much information from that tab which is never enough of the information I require.
5. I then open the website for that business and get the rest of the information that was not in the search tab for one of the search sites.
This is where the automation stop, the rest of the work I do manually.
Start your response with a '$' sign so I know you have actually read the criteria and understand what I am looking for. Please be accurate with your pricing as I want a realistic price to start with.
This is how I do it now.
Throughout this process I manually cut and paste the data I need into a csv file so the company I do work for can check the information. The 3 parts they check is the email addresses, the phone number and the physical address to ensure they are correct and exist.
They send me back the checked data so I know who is worth my time contacting.
My goal is to automate that entire process so I have the completed file to send to the company I do work for and I then can just work on contacting the businesses that check out.
I then take that csv file and convert it into a json file or an sqlite file and do my work from that.
This is why I require different file formats.