Airbnb Scraper
Budget: $1,500 – $3,000 USD
Specs + Scope Airbnb Scraper
Provisions & Expectations
We will provide a repo with a Python scraper code with the ability to query locations by date range and with different specifications. Endpoints for different scrapes are also provided. MongoDB to be provided and the expectation is for this to be uploaded to a GitHub repository and the scraper application should be launched.
Specifications
● Written in Python (Scrapy preferable)
● Must be able to rotate through non-sequential IPs and spoof user agents (evade blocking)
● Build, save and maintain a Listings List
● Build, save and maintain a Markets List
● Summarize Market Scraped Data
● Save Market Summary to MongoDB
● Save Raw Scraped Data in .csv in AWS S3 Bucket
● Continuous scraping scheduled jobs
Scraper
Main Page Search and Capture
Endpoint - https://www.airbnb.com/
The provided scraper code has the ability to query Airbnb’s main page. From the main page search the scraper should be able to search for listings in the markets that will be called from a list provided.
Individual Listings Page
Endpoint - https://www.airbnb.com/rooms/{room_id}
The provided scraper code has the ability to query Airbnb’s individual listings page.
Listing Details
The scraper must collect the following information for each individual listings:
● room_id, name, host_id, photo_count, url, price_rate, price_rate_type, total_price, room_and_property_type, place_id, latitude, longitude, city, state, country, monthly_price_factor, weekly_price_factor, room_type, bedrooms, bathrooms, beds, person_capacity, amenities, review_count, avg_rating, review_score, rating_accuracy, rating_checkin, rating_cleanliness, rating_communication, rating_location, rating_value, star_rating, satisfaction_guest, description, response_rate, response_time, photos, is_hotel, datetime_scrape
Occupancy
Endpoint - https://www.airbnb.com/api/v3/PdpAvailabilityCalendar
The scraper should capture the availability for each day from the current day forward until the end of the availability data present. The scraper should not capture data in this endpoint pertaining to days in the past since it is no longer relevant data.
Pricing
The scraper should capture the price for each day from the current day forward until the end of the availability data present.
Daily Summary by Market
Every scraped day is one summary for a Market that includes all the calendar listing information available in the respective scrape. Listings must be grouped by Markets (Neighborhood, City, State/Region, Country). For every of these categories except Country, all listings information should be grouped in order to summarize the data for the daily summary.
This Daily Summary for Markets will include a total and subcategories attached to it. These subcategories will be based on the bedrooms count for the listings that were scraped:
Example Market in attached file
Provisions & Expectations
We will provide a repo with a Python scraper code with the ability to query locations by date range and with different specifications. Endpoints for different scrapes are also provided. MongoDB to be provided and the expectation is for this to be uploaded to a GitHub repository and the scraper application should be launched.
Specifications
● Written in Python (Scrapy preferable)
● Must be able to rotate through non-sequential IPs and spoof user agents (evade blocking)
● Build, save and maintain a Listings List
● Build, save and maintain a Markets List
● Summarize Market Scraped Data
● Save Market Summary to MongoDB
● Save Raw Scraped Data in .csv in AWS S3 Bucket
● Continuous scraping scheduled jobs
Scraper
Main Page Search and Capture
Endpoint - https://www.airbnb.com/
The provided scraper code has the ability to query Airbnb’s main page. From the main page search the scraper should be able to search for listings in the markets that will be called from a list provided.
Individual Listings Page
Endpoint - https://www.airbnb.com/rooms/{room_id}
The provided scraper code has the ability to query Airbnb’s individual listings page.
Listing Details
The scraper must collect the following information for each individual listings:
● room_id, name, host_id, photo_count, url, price_rate, price_rate_type, total_price, room_and_property_type, place_id, latitude, longitude, city, state, country, monthly_price_factor, weekly_price_factor, room_type, bedrooms, bathrooms, beds, person_capacity, amenities, review_count, avg_rating, review_score, rating_accuracy, rating_checkin, rating_cleanliness, rating_communication, rating_location, rating_value, star_rating, satisfaction_guest, description, response_rate, response_time, photos, is_hotel, datetime_scrape
Occupancy
Endpoint - https://www.airbnb.com/api/v3/PdpAvailabilityCalendar
The scraper should capture the availability for each day from the current day forward until the end of the availability data present. The scraper should not capture data in this endpoint pertaining to days in the past since it is no longer relevant data.
Pricing
The scraper should capture the price for each day from the current day forward until the end of the availability data present.
Daily Summary by Market
Every scraped day is one summary for a Market that includes all the calendar listing information available in the respective scrape. Listings must be grouped by Markets (Neighborhood, City, State/Region, Country). For every of these categories except Country, all listings information should be grouped in order to summarize the data for the daily summary.
This Daily Summary for Markets will include a total and subcategories attached to it. These subcategories will be based on the bedrooms count for the listings that were scraped:
Example Market in attached file