Make changes to existing Python scraping script and set up a VPS to run it
Budget: $10 – $30 USD
We can divide this project into 3 steps.
But please first note:
- I have selected this to be a "Micro project", but bid as you see fit. It must be real bids (not something that will change to be much higher later).
- I will ignore bids without relevant text that makes it clear that you have read this project description.
1: Change an existing working scraping script into a module
--------------------------------------------------------------------------------------------
I have a scraping script that works, but it needs a few small changes so that it instead can be a module that is called by another script. Currently, it reads input from a file and outputs the result to another file.
One change is needed, if possible: The website that is scraped has a login that is valid for 30 minutes. We should keep this connection alive and not re-login each time this is called.
2: Make a main script interacting with database work
---------------------------------------------------------------------------------------
I also have a simple script that does the following:
- Waits for a new record in an input table in Supabase (cloud database)
* If a new record then read the record and also read get the user details from the userDetails table
* Let the scraping module do it's job
* With the result:
1: Update the input record's result field with the result (json)
2: Update the user details if needed (increment a counter)
3: Also send an email via Gmail with the result
- Wait for the next new record...
This last script is not tested, so it might need fixes, and point "2" (counter increment) above is not made.
3: Set up a VPS to work with for these scripts
--------------------------------------------------------------------------
- Make it actually work
- Run only Linux services needed for this to work
- Set up the Firewall to only allow for connection from specific IPs and ports (basically SSH, is anything more needed?)
But please first note:
- I have selected this to be a "Micro project", but bid as you see fit. It must be real bids (not something that will change to be much higher later).
- I will ignore bids without relevant text that makes it clear that you have read this project description.
1: Change an existing working scraping script into a module
--------------------------------------------------------------------------------------------
I have a scraping script that works, but it needs a few small changes so that it instead can be a module that is called by another script. Currently, it reads input from a file and outputs the result to another file.
One change is needed, if possible: The website that is scraped has a login that is valid for 30 minutes. We should keep this connection alive and not re-login each time this is called.
2: Make a main script interacting with database work
---------------------------------------------------------------------------------------
I also have a simple script that does the following:
- Waits for a new record in an input table in Supabase (cloud database)
* If a new record then read the record and also read get the user details from the userDetails table
* Let the scraping module do it's job
* With the result:
1: Update the input record's result field with the result (json)
2: Update the user details if needed (increment a counter)
3: Also send an email via Gmail with the result
- Wait for the next new record...
This last script is not tested, so it might need fixes, and point "2" (counter increment) above is not made.
3: Set up a VPS to work with for these scripts
--------------------------------------------------------------------------
- Make it actually work
- Run only Linux services needed for this to work
- Set up the Firewall to only allow for connection from specific IPs and ports (basically SSH, is anything more needed?)