Fuzzy Duplicate Detection in PostgreSQL

I have a table stored in PostgreSQL that contains a mix of text and numeric fields, and I need a reliable way to spot potential duplicates even when values do not match exactly. Traditional equality checks miss many of the near-matches, so I’m looking for a fuzzy-matching approach—think Levenshtein distance, trigram similarity, or any method you feel is more accurate and performant for this scenario.

The goal is a repeatable solution I can run on demand (SQL script, function, or small Python script that connects via psycopg2 is fine). It should:

• Accept similarity thresholds that I can tweak easily.
• Return the primary keys (or full rows, if simpler) for each suspected duplicate group.
• Be clearly commented so I understand the logic and can extend it later.

If you have prior experience using PostgreSQL’s pg_trgm extension, fuzzystrmatch, or integrating external libraries, mention how you plan to structure the query or code. Once delivered, I’ll test against my current dataset and confirm the matches before we close the project.


My total budget for this project 500INR, you can use any AI tools. start your proposal with PG so i can understand you read it.