ETL project requiring python and pyspark skills
Budget: £10 – £15 GBP
• 5+ years implementing data pipelines or data-intensive assets using Python
• 3+ years using distributed data processing engines such as Apache Spark, Hive
• 2+ years creating modular data transformation using an orchestration engine like airflow or equivalent such as Nifi
• 2+ years building cloud-native solutions in AWS, especially with s3, Glue, Lambda, Step functions, EMR, EC2, or Azure
• Good knowledge on Cloud Databases, especially Snowflake.
• Very good Data analysis skills in SQL. Able to develop complex SQL Statements and tune them for performance when required.
• Experience in the data modelling plan, analysis, design and build effort.
• Experience in ETL framework.
• Understanding of data warehouse and traditional database concepts
• Good interpersonal and communication skills
• 3+ years using distributed data processing engines such as Apache Spark, Hive
• 2+ years creating modular data transformation using an orchestration engine like airflow or equivalent such as Nifi
• 2+ years building cloud-native solutions in AWS, especially with s3, Glue, Lambda, Step functions, EMR, EC2, or Azure
• Good knowledge on Cloud Databases, especially Snowflake.
• Very good Data analysis skills in SQL. Able to develop complex SQL Statements and tune them for performance when required.
• Experience in the data modelling plan, analysis, design and build effort.
• Experience in ETL framework.
• Understanding of data warehouse and traditional database concepts
• Good interpersonal and communication skills