Implementing Spark in Airflow

I am looking for someone who is familiar with both Spark and Airflow. The main goal of implementing Spark in Airflow for my project is to improve scheduling and automation.

Tools: Airflow, Docker, Spark.

Task:

Using Airflow dags, build a pipeline based on distributed computation offered by Spark, but not Pyspark, and keep a log of the pipeline execution and Dockerize it.

1. Download the ETF and stock datasets from the primary dataset available at https://www.kaggle.com/datasets/jacksoncrow/stock-market-dataset.

2. Set up a data structure to retain all data from ETFs and stocks in the following columns.

Symbol: string
Security Name: string
Date: string (YYYY-MM-DD)
Open: float
High: float
Low: float
Close: float
Adj Close: float
Volume: int
Note: Do not change Adj Close to Adj_Close

3.1. Convert the resulting dataset into a structured format (Parquet).
3.2. Calculate the moving average of the trading volume (Volume) of 30 days per each stock and ETF, and retain it in a newly added column vol_moving_avg.
3.3. Similarly, calculate the rolling median and retain it in a newly added column adj_close_rolling_med.
3.4. Retain the resulting dataset into the same format (Parquet).
3.5 write a unit testing.