Assistance with Web Scapping Flight data from a Travel Website with Anti-Bot Protection
Budget: $30 – $250 USD
Project Description
I need assistance in developing a robust web scraping script to extract flight price information from the Smiles website. The goal is to obtain accurate and up-to-date pricing data for analysis.
My current script is already developed, but it's encountering a persistent issue: the website's security tool (Akamai Bot Manager) is blocking API requests with an HTTP 406 - Not Acceptable error. This happens despite sending what appear to be the correct headers and cookies.
The Problem
* The Akamai security system detects that the requests are coming from a bot.
* The server requires dynamic session cookies (_abck, bm_sv) that are generated by complex JavaScript code executed in the browser.
* Solutions based on the Python requests library have failed, as they cannot replicate the full browser environment or generate these cookies with a valid signature.
What We've Already Tried
* Scraping with requests using hardcoded headers and cookies copied from a live browser session.
* We have confirmed that the 406 error occurs because the security cookie token is not being generated correctly.
Ideal Skills and Experience
* Proven experience in bypassing anti-bot systems like Akamai, Cloudflare, or Imperva.
* Proficiency with browser automation tools such as Playwright or Selenium, or specialized scraping APIs designed for these challenges (Scrapfly or ScrapingBee).
* Ability to handle sites with dynamic content and JavaScript.
* Capability to develop a fast and efficient script for multiple searches (e.g., by reusing a single browser session).
* Strong attention to detail to ensure data accuracy.
Note: It's crucial that candidates understand the nature of the Akamai problem and do not propose simple solutions that we have already tried.
I need assistance in developing a robust web scraping script to extract flight price information from the Smiles website. The goal is to obtain accurate and up-to-date pricing data for analysis.
My current script is already developed, but it's encountering a persistent issue: the website's security tool (Akamai Bot Manager) is blocking API requests with an HTTP 406 - Not Acceptable error. This happens despite sending what appear to be the correct headers and cookies.
The Problem
* The Akamai security system detects that the requests are coming from a bot.
* The server requires dynamic session cookies (_abck, bm_sv) that are generated by complex JavaScript code executed in the browser.
* Solutions based on the Python requests library have failed, as they cannot replicate the full browser environment or generate these cookies with a valid signature.
What We've Already Tried
* Scraping with requests using hardcoded headers and cookies copied from a live browser session.
* We have confirmed that the 406 error occurs because the security cookie token is not being generated correctly.
Ideal Skills and Experience
* Proven experience in bypassing anti-bot systems like Akamai, Cloudflare, or Imperva.
* Proficiency with browser automation tools such as Playwright or Selenium, or specialized scraping APIs designed for these challenges (Scrapfly or ScrapingBee).
* Ability to handle sites with dynamic content and JavaScript.
* Capability to develop a fast and efficient script for multiple searches (e.g., by reusing a single browser session).
* Strong attention to detail to ensure data accuracy.
Note: It's crucial that candidates understand the nature of the Akamai problem and do not propose simple solutions that we have already tried.
Related categories:
PHP
JavaScript
Python
Web Scraping
Software Architecture
Scrapy
Data Extraction
Data Analysis
Selenium
API Development