web scraping with Scrapy
Budget: $10 – $30 USD
I am trying to scrap the headline text and dates from this site (https://codeblue.galencentre.org/malaysia/) using Scrapy.
I am stuck at pagination and how to extract the titles from the site.
Although I have edited the user-agent and set Robotxt as False, my attempt that yields 'Error 400:HTTP status code is not handled or not allowed'.
https://pastebin.com/M6mYMzHR
Any ideas on how to handle this? Should I convert 'Content-Type: application/x-www-form-urlencoded' to 'application/json'?
I do not want to use Splash or Selenium
The site is also protected with CloudFlare.
Thank you!
I am stuck at pagination and how to extract the titles from the site.
Although I have edited the user-agent and set Robotxt as False, my attempt that yields 'Error 400:HTTP status code is not handled or not allowed'.
https://pastebin.com/M6mYMzHR
Any ideas on how to handle this? Should I convert 'Content-Type: application/x-www-form-urlencoded' to 'application/json'?
I do not want to use Splash or Selenium
The site is also protected with CloudFlare.
Thank you!