How to avoid robot judgment for site scraping
Budget: $30 – $250 USD
If you access the site and click the check box to search, it will be judged as a robot.The checkbox click uses selenium's chromedriver.What I want to do is click the checkbox and transition to the next page.
The execution environment is python3.10, selenium.
Here is the script.
-------------------------------------------------------
from selenium import webdriver
import time
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome(options=ChromeOptions)
driver.implicitly_wait(20)
url = 'https://www.rakumachi.jp/syuuekibukken/area/prefecture/dimAll/'
driver.get(url)
time.sleep(1)
driver.execute_script("window.scrollTo(0, 1200);")
# click1
driver.find_element("xpath", "//div[@class='searchItem']/ul[@class='horizonal_list w22']//li[11]/label").click()
# click2
driver.find_element("xpath", "//input[@id='form_submit_button']").click()
# robot judgment
-------------------------------------------------------
The execution environment is python3.10, selenium.
Here is the script.
-------------------------------------------------------
from selenium import webdriver
import time
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome(options=ChromeOptions)
driver.implicitly_wait(20)
url = 'https://www.rakumachi.jp/syuuekibukken/area/prefecture/dimAll/'
driver.get(url)
time.sleep(1)
driver.execute_script("window.scrollTo(0, 1200);")
# click1
driver.find_element("xpath", "//div[@class='searchItem']/ul[@class='horizonal_list w22']//li[11]/label").click()
# click2
driver.find_element("xpath", "//input[@id='form_submit_button']").click()
# robot judgment
-------------------------------------------------------