Python JSON Noise Cleanup
Budget: $30 – $250 USD
I have a collection of JSON files that need to be sanitized. The main issues are duplicate entries and occasional formatting problems that break downstream tooling. A record is a duplicate when the value of its ID field matches an earlier one; keep the first instance and discard the rest. After deduplication, the script must also detect and correct—or clearly flag—any invalid data formats so that every file remains valid JSON and passes a simple load/parse test.
Please deliver a self-contained Python script (Python 3.x) that:
• Accepts a folder path or list of files as input and processes them in-place or to a specified output directory.
• Removes duplicates based solely on the ID field.
• Validates each object’s structure, fixing minor issues such as stray commas or unescaped characters when feasible; otherwise, move the problematic records to a separate “rejected.json” report.
• Prints a short summary (number of files scanned, duplicates removed, records fixed/moved).
Acceptance will be based on running the script against a sample set I provide: all duplicates gone, no malformed JSON left behind, and the summary matching manual counts. If you can keep the design modular so that other fields (Name, Email, etc.) could be swapped in later, that’s a bonus but not required.
Please deliver a self-contained Python script (Python 3.x) that:
• Accepts a folder path or list of files as input and processes them in-place or to a specified output directory.
• Removes duplicates based solely on the ID field.
• Validates each object’s structure, fixing minor issues such as stray commas or unescaped characters when feasible; otherwise, move the problematic records to a separate “rejected.json” report.
• Prints a short summary (number of files scanned, duplicates removed, records fixed/moved).
Acceptance will be based on running the script against a sample set I provide: all duplicates gone, no malformed JSON left behind, and the summary matching manual counts. If you can keep the design modular so that other fields (Name, Email, etc.) could be swapped in later, that’s a bonus but not required.