Assistance with using Python to automate a cookie issue with web scrape task
Budget: $10 – $30 USD
Overall project goal: Automate the scraping of numerical values, with corresponding dates, from the first and second last charts from the linked website.
Goal for this job: Using Python, generate/create/fetch the ‘x-rap-xsrf-token’ (and cookie, if need be) (which expires every few hours) prior to the POST request to a website.
We want to run a script that scrapes the numerical values from these charts, but we need this x-rap-xsrf-token in the POST request header.
What we have done so far: Copied the whole Request Payload JSON file for the respective POST requests from Chrome’s Network tab. Fetched relevant header information like “accept”, “content-type”, “user-agent”, “cookie”, “x-rap-xsrf-token” manually.
What we need help with: The x-rap-xsrf-token and cookie seem to update every few hours. This ‘manual’ part is the only thing we need automated to fully automate our scraping process. Right now, we manually update the hard-coded x-rap-xsrf-token (copy-pasted from Networks tab in Chrome) in the request header. Seems like we need the ‘cookie’ item in the request header, but it doesn’t need to be updated each time.
We will send through the code which has been completed thus far. Your only task is to automate the last step of the project. Thank you!
Goal for this job: Using Python, generate/create/fetch the ‘x-rap-xsrf-token’ (and cookie, if need be) (which expires every few hours) prior to the POST request to a website.
We want to run a script that scrapes the numerical values from these charts, but we need this x-rap-xsrf-token in the POST request header.
What we have done so far: Copied the whole Request Payload JSON file for the respective POST requests from Chrome’s Network tab. Fetched relevant header information like “accept”, “content-type”, “user-agent”, “cookie”, “x-rap-xsrf-token” manually.
What we need help with: The x-rap-xsrf-token and cookie seem to update every few hours. This ‘manual’ part is the only thing we need automated to fully automate our scraping process. Right now, we manually update the hard-coded x-rap-xsrf-token (copy-pasted from Networks tab in Chrome) in the request header. Seems like we need the ‘cookie’ item in the request header, but it doesn’t need to be updated each time.
We will send through the code which has been completed thus far. Your only task is to automate the last step of the project. Thank you!