Simple data Pipeline with airflow and spark in docker container
Budget: £10 – £20 GBP
I want to develop a pipeline in docker using airflow and spark, where source data is located on a folder that is mounted on all containers ( I have started the project already. I have a zipped version of it)
task1 = check if the source file (csv) exists using bash operator (I have done that part)
task 2 = using spark=submit or bashoperator , run the pyspark file(provided) that loads the source data and transforms it and also saves to a parquet format located in the same directory of the source file
task 3 = load the paquet format and run sql query with csv as output
task 4 - rename this output file by adding todays date
task 5 : push the file into a gmail account
task 6 : change the file name of the source data to avaoid errors when a new file lands in the soucre data folder
task1 = check if the source file (csv) exists using bash operator (I have done that part)
task 2 = using spark=submit or bashoperator , run the pyspark file(provided) that loads the source data and transforms it and also saves to a parquet format located in the same directory of the source file
task 3 = load the paquet format and run sql query with csv as output
task 4 - rename this output file by adding todays date
task 5 : push the file into a gmail account
task 6 : change the file name of the source data to avaoid errors when a new file lands in the soucre data folder