AWS Engineer / ETL Developer
Budget: $5,000 – $10,000 USD
Skillset Needed:
Defect resolution and production support of Big data ETL development using AWS native services
Create data pipeline architecture by designing and implementing data ingestion solutions
Integrate data sets using AWS services such as Glue, Lambda functions
Design and optimize data models on AWS Cloud using AWS data stores such as Redshift, RDS, S3, Athena
Author ETL processes using Python, Pyspark
ETL process monitoring using Cloudwatch events
You will be working in collaboration with other teams
We are looking for a engineer to resolve these issues described below in our AWS environment.
Enable paging through data returned from each API using the offset field.
Delta Load enablement for Dimension tables (16), Fact tables(6), and Derived Tables(4)
Go back in time and reprocess Fact tables(6), and Derived Tables(4) from the said date automatically (retro mode)
Dimension tables such as entity_device, entity_assets taking longer time (~70-80 minutes) to load - need performance improvement to raw data glue job.
Defect resolution and production support of Big data ETL development using AWS native services
Create data pipeline architecture by designing and implementing data ingestion solutions
Integrate data sets using AWS services such as Glue, Lambda functions
Design and optimize data models on AWS Cloud using AWS data stores such as Redshift, RDS, S3, Athena
Author ETL processes using Python, Pyspark
ETL process monitoring using Cloudwatch events
You will be working in collaboration with other teams
We are looking for a engineer to resolve these issues described below in our AWS environment.
Enable paging through data returned from each API using the offset field.
Delta Load enablement for Dimension tables (16), Fact tables(6), and Derived Tables(4)
Go back in time and reprocess Fact tables(6), and Derived Tables(4) from the said date automatically (retro mode)
Dimension tables such as entity_device, entity_assets taking longer time (~70-80 minutes) to load - need performance improvement to raw data glue job.