Data Governance Policy ML Python POC
Budget: $30 – $250 USD
I have ~2000 individual users and metadata about them e.g. location, cost_centre rollup etc. I have some tables which have hierarchies i.e. cost centre hierarchy with parent-child relationships.
I have ~1500 existing row filters, applied by "group" applied to a table in order to filter the data they see. I have a mapping of individual : "group".
About 50% of users only have access to 1 group, about 90% of users have access to 3 groups or less. Therefore, there is a strong chance that we can create a row filter based off the users metadata directly, instead of going through complicated ETL's and processing.
I want to compare the metadata to the existing row filters, determine the significant factors, and generate a row policy "just as good" as the existing row filters. I do currently generate and apply the ~1500 policies, so I can explain how, so we can reuse some of this logic too.
I have a layout already and some code done you can build off of. I would prefer screenshare and we code together so we can test things as we go, but I am flexible.
Deliverable:
Python notebook or script that loops through users and does the following:
1) Generates a row filter based on the user metadata only
2) Compares a count(*) on a table input of the row filter vs the prod filter
3) Iterate to improve the model to make it more accurate
We only need to make it optimal for 1 table as this is a proof of concept deliverable.
More work is available for a good deliverable.
I have ~1500 existing row filters, applied by "group" applied to a table in order to filter the data they see. I have a mapping of individual : "group".
About 50% of users only have access to 1 group, about 90% of users have access to 3 groups or less. Therefore, there is a strong chance that we can create a row filter based off the users metadata directly, instead of going through complicated ETL's and processing.
I want to compare the metadata to the existing row filters, determine the significant factors, and generate a row policy "just as good" as the existing row filters. I do currently generate and apply the ~1500 policies, so I can explain how, so we can reuse some of this logic too.
I have a layout already and some code done you can build off of. I would prefer screenshare and we code together so we can test things as we go, but I am flexible.
Deliverable:
Python notebook or script that loops through users and does the following:
1) Generates a row filter based on the user metadata only
2) Compares a count(*) on a table input of the row filter vs the prod filter
3) Iterate to improve the model to make it more accurate
We only need to make it optimal for 1 table as this is a proof of concept deliverable.
More work is available for a good deliverable.