Azure: Customize data factory for delta logic
Budget: €8 – €30 EUR
Two new columns are required in the Postgres database for this purpose:
Datahash
LastModified
A hash is stored in Datahash, which is calculated from all columns except for Datahash and LastModified.
When transferring the data from Conceptum, the method should be adjusted:
Calculate hash from new data
Compare new hash with old
If matching, data hash, and LastModified do not change
Otherwise, insert a new hash into the data hash and set LastModified to the current time (now).
Use new data from Conceptum
Datahash
LastModified
A hash is stored in Datahash, which is calculated from all columns except for Datahash and LastModified.
When transferring the data from Conceptum, the method should be adjusted:
Calculate hash from new data
Compare new hash with old
If matching, data hash, and LastModified do not change
Otherwise, insert a new hash into the data hash and set LastModified to the current time (now).
Use new data from Conceptum