Process large JSON (100GB+ file) using JSON streaming
Budget: $250 – $750 USD
I'm looking for someone to write a Node/JS script that will stream 3 JSON files (each about 100Gb in size) concurrently, match the items from those streams based on some key-value pair in the JSON, and then write the result to a DB.
This needs to be a high performance script. I have written some code myself, but it is suffering from memory leaks and memory allocation errors due to the large file size. You are free to use whatever library you have available.
This needs to be a high performance script. I have written some code myself, but it is suffering from memory leaks and memory allocation errors due to the large file size. You are free to use whatever library you have available.