Airflow Daily Python Job

Job ID: 30889705

Budget: $15 – $40 USD

-- Fixed Price Task --

1. Schedule a daily job in Airflow that store files in database
* Implement DAG/Orchestration functions to schedule the job in Airflow.
* Use the existing python attached and make the necessary changes to the code to implement the job in Airflow

2. Today, the existing python code creates and stores the web-scrapped files into different folders and also inside a dataframe
- Keep that logic but implement also the logic, where:
- It stores the files into a database (Postgree)
- check first if the file exists in the database
- if it doesn't, than store the new record

** If you don't have Airflow environment installed
- we advise to put the environment up in docker
- simply docker-compose up from the official build here in the official website
https://github.com/apache/airflow/blob/main/docs/apache-airflow/start/docker-compose.yaml

Closed scope task
** You will be responsible to guide our team on Airflow deployment in the end **
*** Partials executions of this task wont be accepted
**** Please don't bid if you cant execute it from beginning to the end.

-> Python existing Code already attached here
-> if needed Airflow manual step-by-step guide attached
(Bid your price within max defined budget range)