Web Scraper Project
Budget: $275 – $276 USD
I need a proficient web scraper to extract all records from the New York State Unified Court System website (https://iapps.courts.state.ny.us/attorneyservices/search?3). The task involves iterating through various name combinations to access every individual profile and collect all of the corresponding fields from all of the profiles (some fields may be empty).
There is a No CAPTCHA reCAPTCHA that surfaces approximately once every 10 new searches. (A No CAPTCHA reCAPTCHA is a specific type of reCAPTCHA that requires users to simply click a checkbox labeled "I am human" to complete the verification process.)
No negotiations on price.
Requirements:
Name Iteration: The scraper must cycle through all name options – using one letter of the first name and two letters of the last name – to access every individual profile. (For example, the first 4 searches would be: firstname = “a”, lastname = “aa”; firstname = “b”, lastname = “aa”; firstname = “c”, lastname = “aa”; firstname = “d”, lastname = “aa”. The first 26 would continue on in this pattern, then the 27th would be firstname = “a”, lastname = “ab” and would continue on in this manner until firstname = “z” and lastname = “zz”. Obviously, some searches have no results, but most will. To be clear, this requires collecting the records of each 17,576 searches. Most searches will have multiple results. Each field of each record from each search needs to be collected.)
Profile Access: The scraper MUST click on each name, as the fields are only available on the individual profile page.
Fields that MUST be extracted: Registration Number, Name, Business Name, Business Address, Business Phone, Email, Date Admitted, Appellate Division Department of Admission, Law School, Registration Status, Next Registration.
Data Format: The extracted data should be delivered in CSV format, with each field separated by a comma.
I would like 4 files. The first file should contain All records with last names that begin with “A” through “E”, the second should have all last names from “F” through “K”, the third will have all last names from “L” through “R”, and the fourth will have all last names from “S” through “Z”.
The ideal candidate for this project should have:
- Extensive experience in web scraping
- Proficiency in data filtering and organization
- Ability to deliver data in a specified format
No negotiations on price.
There is a No CAPTCHA reCAPTCHA that surfaces approximately once every 10 new searches. (A No CAPTCHA reCAPTCHA is a specific type of reCAPTCHA that requires users to simply click a checkbox labeled "I am human" to complete the verification process.)
No negotiations on price.
Requirements:
Name Iteration: The scraper must cycle through all name options – using one letter of the first name and two letters of the last name – to access every individual profile. (For example, the first 4 searches would be: firstname = “a”, lastname = “aa”; firstname = “b”, lastname = “aa”; firstname = “c”, lastname = “aa”; firstname = “d”, lastname = “aa”. The first 26 would continue on in this pattern, then the 27th would be firstname = “a”, lastname = “ab” and would continue on in this manner until firstname = “z” and lastname = “zz”. Obviously, some searches have no results, but most will. To be clear, this requires collecting the records of each 17,576 searches. Most searches will have multiple results. Each field of each record from each search needs to be collected.)
Profile Access: The scraper MUST click on each name, as the fields are only available on the individual profile page.
Fields that MUST be extracted: Registration Number, Name, Business Name, Business Address, Business Phone, Email, Date Admitted, Appellate Division Department of Admission, Law School, Registration Status, Next Registration.
Data Format: The extracted data should be delivered in CSV format, with each field separated by a comma.
I would like 4 files. The first file should contain All records with last names that begin with “A” through “E”, the second should have all last names from “F” through “K”, the third will have all last names from “L” through “R”, and the fourth will have all last names from “S” through “Z”.
The ideal candidate for this project should have:
- Extensive experience in web scraping
- Proficiency in data filtering and organization
- Ability to deliver data in a specified format
No negotiations on price.