Airflow xComms Dag

Job ID: 35621573

Budget: €18 – €36 EUR

Please apply only if you have strong experience in Airflow xComm.
Using Airflow 2.5 we would like to have a task to query a Postgres table and pass the result of this query to another task (Python Task) which matches the columns obtained from the Postgres Task with a configuration file and passes the output to a third task, which iterates through the result set. This all will likely need to be done with xComm.

For reference: https://docs.astronomer.io/learn/dynamic-tasks

In Summary, the Dag should:

Get the Config file from a directory
From the config file create a list if SQLs
Run each SQL in Postgres and return a result (Postgres dynamic tasks created from 3)
From the result of each SQL create an Airflow python dynamic task

I need help with a DAG that will have 2 tasks:
1. query data from Postgres DB
2. pass the results of the data to another task that will circle through the results and pass them to the 3rd python task

Experience with Airflow 2.5 and xComms is mandatory. Only apply if you have done this and know what is xcomm and how to pass variables between tasks
Related categories: Python PostgreSQL