Need someone who can write a regression script to validate data.
Budget: ₹1,500 – ₹12,500 INR
I am working on Data Migration(migrating to AWS). I need a regression test script that can be used in Data Ingestion Validations so that all the developers can run it for the data loaded by Datastage(EDS) vs Lakes ETL process(ESDL). We have 2 databases in Snoflake; EDS and ESDL and we are migrating data to ESDL using AWS services. The script should be able to take input schema/table_name and a "where clause" to restrict data to a certain period and also include an "exclude column list" using which, it should ignore those columns in the data compares. Eg: Most of the tables I work on have record_created_timestamp column. So, I would want that to be excluded when I do data comparisons between EDS(Datastage) and ESDL(Lakes ETL process) as it is most likely to have different values. Also, I want that it could run in AWS Glue job integrated with SNS topic and orchestrated by Step function or Lambda Integrated with SNS topic.
3 files are attached.
1st file: It is a code we used previously to compare data Greenplum and Snowflake. I hope it helps.
2nd file: It is a AWS Glue job script stating how we fetch data from Snowflake.
3rd file: A word file that has a sample of the data I work on.
A basic overview would be needed at the end of this task. Help would be appreciated.
3 files are attached.
1st file: It is a code we used previously to compare data Greenplum and Snowflake. I hope it helps.
2nd file: It is a AWS Glue job script stating how we fetch data from Snowflake.
3rd file: A word file that has a sample of the data I work on.
A basic overview would be needed at the end of this task. Help would be appreciated.