Building airflow custom hooks/operators for cloud
Budget: $30 – $250 SGD
To build airflow hooks and operators for 2 cloud services - object storage and data warehouse.
Required programming language: Python
Airflow plug-ins documentation: 1. https://airflow.apache.org/docs/apache-airflow/stable/concepts/connections.html
2. https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html
Guide: https://medium.com/b6-engineering/airflow-writing-your-own-operators-and-hooks-93fcfbc7bd
Cloud Services:
1. OBS (object storage) - from MySQL to OBS and from MongoDB to OBS
OBS python SDK - https://support.huaweicloud.com/intl/en-us/sdk-python-devg-obs/obs_22_0002.html
2. DWS (Data warehouse) - from OBS to DWS or DWS to OBS
DWS API documentation: https://support.huaweicloud.com/intl/en-us/api-dws/dws_02_0033.html
Required programming language: Python
Airflow plug-ins documentation: 1. https://airflow.apache.org/docs/apache-airflow/stable/concepts/connections.html
2. https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html
Guide: https://medium.com/b6-engineering/airflow-writing-your-own-operators-and-hooks-93fcfbc7bd
Cloud Services:
1. OBS (object storage) - from MySQL to OBS and from MongoDB to OBS
OBS python SDK - https://support.huaweicloud.com/intl/en-us/sdk-python-devg-obs/obs_22_0002.html
2. DWS (Data warehouse) - from OBS to DWS or DWS to OBS
DWS API documentation: https://support.huaweicloud.com/intl/en-us/api-dws/dws_02_0033.html