Azure data bricks - Auditing and Logging POC- Rule based data validation checks for file processing

Its for Azure ETL data warehousing audit task
In audit task - need to manage data validation checks
But these data validation checks should be able to operate from Azure data lake file
i.e. Rule engine will get operated via external table structure
i.e Rule name and code will present in Azure data lake and execution will happen within Azure data bricks notebook

That means:

There will be a master table-
1)1st column - it will contain Data validation rule name
2)2nd column - it will contain code which will be applied for named validation checks
(such that this code value will be called within variable and from there function is created which will be applied against records in data bricks)

Now, every time data bricks will execute a file, it will apply data validation checks against required fields


Then there will be a respective logging table -

So it will maintain all records entry along with that files name if it has any specific data validation check failure

So its small scale data auditing and error logging POC for data validation checks like email and phone number.
Technology Used - Azure Data bricks
Tables needs to be created as - Azure data lake files


Rule Engine:
1. All the Rules will be saved in a table in Data Lake (as of now rule are only for email and phone number validation checks)
2. Rules will be executed on all the incoming file data
3. Failed rows will be placed in a separate table with the reason of failure


PFA to get a brief idea about what kind of azure data lake table structure we are requesting. Thank you.