Real-Time Scraper API for Copart & IAAI
Budget: $30 – $250 USD
I'm looking for a professional data engineer to create a robust API for real-time data extraction from Copart and IAAI auctions.
Key Requirements:
- Scrape comprehensive car details including make, model, and year
- Collect auction specifics such as location, date, and time
- Track and record bidding history
The ideal freelancer should have a strong background in web scraping, API development, and data analysis. Experience with auction data and familiarity with Copart and IAAI platforms will be a significant advantage.
The goal of this project is to develop a scraper API to extract live and historical auction data from the Copart and IAAI websites. Both websites are protected by Cloudflare, so the API must be able to bypass Cloudflare and handle large volumes of requests (approximately 1 million per day). The API will scrape auction details, including text data and images, and store this data in PostgreSQL as well as in files.
The API must be able to function in a Docker environment and integrate well with PostgreSQL. The final product will be tested on at least 1,000,000 records.
Key Requirements:
- Scrape comprehensive car details including make, model, and year
- Collect auction specifics such as location, date, and time
- Track and record bidding history
The ideal freelancer should have a strong background in web scraping, API development, and data analysis. Experience with auction data and familiarity with Copart and IAAI platforms will be a significant advantage.
The goal of this project is to develop a scraper API to extract live and historical auction data from the Copart and IAAI websites. Both websites are protected by Cloudflare, so the API must be able to bypass Cloudflare and handle large volumes of requests (approximately 1 million per day). The API will scrape auction details, including text data and images, and store this data in PostgreSQL as well as in files.
The API must be able to function in a Docker environment and integrate well with PostgreSQL. The final product will be tested on at least 1,000,000 records.