Python Web Scraper for Data Collection (Eastern Europe or Latin America)
Budget: $25 – $100 USD
*******
LOCATION
*******
You must live in Eastern Europe or Latin America.
*******
OVERVIEW
*******
I will conduct searches on Google like "Plumbers Minneapolis, Minnesota" or "Carpenters Charlotte, North Carolina."
Then I will run the script(s) you created for me.
The script(s) you create will save the Google search results in a .csv file. That is trivial and straightforward. The script(s) will also open the websites associated with the search results. That is also trivial and straightforward. The main challenge lies in locating and filling out contact forms on these websites with my first name, last name, email address, phone number, etc.
Most search results will include a link to the company’s website. For example, when I search on Google for… Jake The Plumber Minneapolis, MN … I see a link to Jake The Plumber’s website.
Your script(s) will not submit the forms; rather, I will review each form and then manually submit each myself. In other words, I will click on, for example, the “Send” or “Submit” button.
If I were to loop your script(s) X times, then your script(s) would open X tabs in Google Chrome. Each tab would contain a website for a particular business.
For example, if I were to loop your script(s) 89 times, then your script(s) would open 89 tabs in Google Chrome. Each tab would contain a website for a particular business.
Most such websites I have visited take about 100 MB of RAM. I have 16 GB of RAM. If I were to free up 12 GB of RAM, then I could open [takes out pencil and paper, calculates furiously for a few minutes] 120 websites.
That limitation would not bother me because I plan to scrape not more than a hundred pages each day. In other words, I do not plan to scrape hundreds of thousands, nor tens of thousands, nor even thousands of websites each day. I plan to use your scripts on a relatively small scale of not more than one hundred websites each day.
*******
TESTING
*******
You must test your script on Linux Mint 21.3 Cinnamon Edition to ensure that it runs properly on Python 3.10.12, on the current stable version of Google Chrome, on Linux Mint 21.3 Cinnamon Edition. I do not care if your script runs on any other operating systems or any other browsers.
If you typically use Microsoft Windows, please consider setting up a virtual machine to install Linux Mint 21.3 Cinnamon Edition so you can test your code properly.
In particular, if I were to send you a bug report, I would not like to receive a reply from you like, “The script works 100% perfectly for me on Google Chrome on Windows 11. Because I am a 100% perfect engineer, who always ships code that is 100% bug-free, I am 100% certain that the code is 100% cross-platform compatible; therefore, I am 100% certain that it is your fault that the script is not running properly.”
Of course I am being facetious. No engineer I've ever worked with has explicitly made such ludicrous statements. Nonetheless, I want you to understand the frustration I have had dealing with incompetent, self-professed software engineers who have shipped me buggy code which they failed to test on Linux, and then blamed me.
*******
SPECIFICATIONS
*******
I will send you complete specifications for this project.
*******
A MAIN SCRIPT THAT EXECUTES SUBORDINATE SCRIPTS
*******
To make it easier to modify (in particular, to debug) you will provide me with a main script named “Main script for scraping and filling in a contact form.py”
that executes subordinate scripts. For example, “Main script for scraping and filling in a contact form.py” might contain…
import subprocess
# Common base directory
base_dir = "~/abc/xyz/"
# Run script 1
subprocess.run(["python3", base_dir + "1 of 3 script for scraping and filling in a contact form.py"])
# Run script 2
subprocess.run(["python3", base_dir + "2 of 3 script for scraping and filling in a contact form.py"])
# Run script 3
subprocess.run(["python3", base_dir + "3 of 3 script for scraping and filling in a contact form.py"])
In this example, almost all of your code would have been in…
"1 of 3 script for scraping and filling in a contact form.py", and
"2 of 3 script for scraping and filling in a contact form.py", and
"3 of 3 script for scraping and filling in a contact form.py"
*******
THE NUMBER OF ITERATIONS AND THE LENGTH OF THE PAUSES
*******
I don’t want a Google algorithm to ban me for web scraping. Therefore, you will give me a script named, “Trigger main script for scraping and filling in a contact form.py” which will include something like the following…
import subprocess, random, time
# Define the Python script file to be executed
script_file = "Main script for scraping and filling in a contact form.py"
# Define the number of times to loop
number_of_iterations = 71
# Define the range for random interval (between X and Y seconds)
min_interval = 10 # Minimum interval in seconds
max_interval = 120 # Maximum interval in seconds
# Loop to execute the script multiple times with a random delay
for _ in range(number_of_iterations):
# Generate a random delay between min_interval and max_interval seconds
random_delay = random.uniform(min_interval, max_interval)
# Execute the script
subprocess.run(["python", script_file])
# Pause execution for the random delay
time.sleep(random_delay)
Obviously, this will enable me to manually change the number of iterations, and the length of the pause between each iteration.
You don’t have to use the precise code above; but your script(s) must contain the functionality above.
*******
QUESTION
*******
Which do you intend to use: Playwright or Selenium?
*******
FINAL NOTE
*******
Please do not begin working on this project for me unless I create a Milestone Payment for you on Freelancer.
LOCATION
*******
You must live in Eastern Europe or Latin America.
*******
OVERVIEW
*******
I will conduct searches on Google like "Plumbers Minneapolis, Minnesota" or "Carpenters Charlotte, North Carolina."
Then I will run the script(s) you created for me.
The script(s) you create will save the Google search results in a .csv file. That is trivial and straightforward. The script(s) will also open the websites associated with the search results. That is also trivial and straightforward. The main challenge lies in locating and filling out contact forms on these websites with my first name, last name, email address, phone number, etc.
Most search results will include a link to the company’s website. For example, when I search on Google for… Jake The Plumber Minneapolis, MN … I see a link to Jake The Plumber’s website.
Your script(s) will not submit the forms; rather, I will review each form and then manually submit each myself. In other words, I will click on, for example, the “Send” or “Submit” button.
If I were to loop your script(s) X times, then your script(s) would open X tabs in Google Chrome. Each tab would contain a website for a particular business.
For example, if I were to loop your script(s) 89 times, then your script(s) would open 89 tabs in Google Chrome. Each tab would contain a website for a particular business.
Most such websites I have visited take about 100 MB of RAM. I have 16 GB of RAM. If I were to free up 12 GB of RAM, then I could open [takes out pencil and paper, calculates furiously for a few minutes] 120 websites.
That limitation would not bother me because I plan to scrape not more than a hundred pages each day. In other words, I do not plan to scrape hundreds of thousands, nor tens of thousands, nor even thousands of websites each day. I plan to use your scripts on a relatively small scale of not more than one hundred websites each day.
*******
TESTING
*******
You must test your script on Linux Mint 21.3 Cinnamon Edition to ensure that it runs properly on Python 3.10.12, on the current stable version of Google Chrome, on Linux Mint 21.3 Cinnamon Edition. I do not care if your script runs on any other operating systems or any other browsers.
If you typically use Microsoft Windows, please consider setting up a virtual machine to install Linux Mint 21.3 Cinnamon Edition so you can test your code properly.
In particular, if I were to send you a bug report, I would not like to receive a reply from you like, “The script works 100% perfectly for me on Google Chrome on Windows 11. Because I am a 100% perfect engineer, who always ships code that is 100% bug-free, I am 100% certain that the code is 100% cross-platform compatible; therefore, I am 100% certain that it is your fault that the script is not running properly.”
Of course I am being facetious. No engineer I've ever worked with has explicitly made such ludicrous statements. Nonetheless, I want you to understand the frustration I have had dealing with incompetent, self-professed software engineers who have shipped me buggy code which they failed to test on Linux, and then blamed me.
*******
SPECIFICATIONS
*******
I will send you complete specifications for this project.
*******
A MAIN SCRIPT THAT EXECUTES SUBORDINATE SCRIPTS
*******
To make it easier to modify (in particular, to debug) you will provide me with a main script named “Main script for scraping and filling in a contact form.py”
that executes subordinate scripts. For example, “Main script for scraping and filling in a contact form.py” might contain…
import subprocess
# Common base directory
base_dir = "~/abc/xyz/"
# Run script 1
subprocess.run(["python3", base_dir + "1 of 3 script for scraping and filling in a contact form.py"])
# Run script 2
subprocess.run(["python3", base_dir + "2 of 3 script for scraping and filling in a contact form.py"])
# Run script 3
subprocess.run(["python3", base_dir + "3 of 3 script for scraping and filling in a contact form.py"])
In this example, almost all of your code would have been in…
"1 of 3 script for scraping and filling in a contact form.py", and
"2 of 3 script for scraping and filling in a contact form.py", and
"3 of 3 script for scraping and filling in a contact form.py"
*******
THE NUMBER OF ITERATIONS AND THE LENGTH OF THE PAUSES
*******
I don’t want a Google algorithm to ban me for web scraping. Therefore, you will give me a script named, “Trigger main script for scraping and filling in a contact form.py” which will include something like the following…
import subprocess, random, time
# Define the Python script file to be executed
script_file = "Main script for scraping and filling in a contact form.py"
# Define the number of times to loop
number_of_iterations = 71
# Define the range for random interval (between X and Y seconds)
min_interval = 10 # Minimum interval in seconds
max_interval = 120 # Maximum interval in seconds
# Loop to execute the script multiple times with a random delay
for _ in range(number_of_iterations):
# Generate a random delay between min_interval and max_interval seconds
random_delay = random.uniform(min_interval, max_interval)
# Execute the script
subprocess.run(["python", script_file])
# Pause execution for the random delay
time.sleep(random_delay)
Obviously, this will enable me to manually change the number of iterations, and the length of the pause between each iteration.
You don’t have to use the precise code above; but your script(s) must contain the functionality above.
*******
QUESTION
*******
Which do you intend to use: Playwright or Selenium?
*******
FINAL NOTE
*******
Please do not begin working on this project for me unless I create a Milestone Payment for you on Freelancer.