Data Engineer Task -- 2
Budget: €8 – €30 EUR
Task description
Prepare a short presentation of about 15-20 minutes where you describe and evaluate your solution. You may implement your solution with the GCP free tier offering to get a better understanding, and use this as part of your presentation and demonstrate your implementation.
Context
Our customer has an Apache Hadoop Instance with a HDFS backend in GCP. He needs to transfer deduplicated datasets to GCP's BigQuery Service. Currently, this customer uses a custom dockerized Python script to achieve this task. Since there are several similar tasks upcoming, this customer decided that the cumbersome and expansive undertaking of maintaining dozens of dockerized python scripts should be superseded by a more general solution, which is cost effective, fast and easier to maintain.
Solution
It's our goal to present GCP Composer - GCP managed Apache Airflow - as a general solution for such tasks which offers said flexibility, performance, and cost effectiveness.
Please build a basic Composer flow (or in Airflow naked)
Bonus: In your DAG, please generate a new possibly interesting column ( “feature generation”)
Please prepare a high level presentation (2-3 slides enough about flow architecture and what the data is about)
Make some quick sanity / quality checks of a few columns (e.g. their distribution)
Dataset
You may use these MusciBrainz Datasets and deduplicate them. (Hint: If the CID is identical, they are duplicates. Assume that this ID is available to you, so you can use it to deduplicate the data easily.)
Important Note
If you have no experience with Composer / Airflow, try Composer first, as you don’t have to install it locally – which is often tedious for Windows Computers. Besides, use a GCP Voucher and choose cheap computer.
Data set link:
https://vsis-www.informatik.uni-hamburg.de/oldServer/teaching/projects/QloUD/DaPo/testdata/
Please explain me overview of the solution so that I may award task confidently.
Prepare a short presentation of about 15-20 minutes where you describe and evaluate your solution. You may implement your solution with the GCP free tier offering to get a better understanding, and use this as part of your presentation and demonstrate your implementation.
Context
Our customer has an Apache Hadoop Instance with a HDFS backend in GCP. He needs to transfer deduplicated datasets to GCP's BigQuery Service. Currently, this customer uses a custom dockerized Python script to achieve this task. Since there are several similar tasks upcoming, this customer decided that the cumbersome and expansive undertaking of maintaining dozens of dockerized python scripts should be superseded by a more general solution, which is cost effective, fast and easier to maintain.
Solution
It's our goal to present GCP Composer - GCP managed Apache Airflow - as a general solution for such tasks which offers said flexibility, performance, and cost effectiveness.
Please build a basic Composer flow (or in Airflow naked)
Bonus: In your DAG, please generate a new possibly interesting column ( “feature generation”)
Please prepare a high level presentation (2-3 slides enough about flow architecture and what the data is about)
Make some quick sanity / quality checks of a few columns (e.g. their distribution)
Dataset
You may use these MusciBrainz Datasets and deduplicate them. (Hint: If the CID is identical, they are duplicates. Assume that this ID is available to you, so you can use it to deduplicate the data easily.)
Important Note
If you have no experience with Composer / Airflow, try Composer first, as you don’t have to install it locally – which is often tedious for Windows Computers. Besides, use a GCP Voucher and choose cheap computer.
Data set link:
https://vsis-www.informatik.uni-hamburg.de/oldServer/teaching/projects/QloUD/DaPo/testdata/
Please explain me overview of the solution so that I may award task confidently.