PostgreSQL help with mass data update
Budget: $10 – $30 USD
I am new to PostgreSQL and i need someone to speed up some work for me.
I have about 1 billion records to modify.
I need to clean special characters and accents, example É is changed to E.
At the moment I have a php program that loops through the 1 billion records in chunks and checks each record to see if there are special characters. If it finds special characters, it replaces them and then performs an update to the record.
This process is running at a rate of about 6 million records per day, which is too slow for me.
I have tried running the program on in-house fast servers and cloud-virtual servers and the speed is always the same, at about 6 million per 24 hours.
I understand that each record must be checked, and that will take time, but I am wondering if there is any trick to making this run faster, because at the current rate, it will take too long.
If you are an expert with PostgreSQL and are able to provide me with a solution that will make this work faster, then please contact me.
Thank you
I have about 1 billion records to modify.
I need to clean special characters and accents, example É is changed to E.
At the moment I have a php program that loops through the 1 billion records in chunks and checks each record to see if there are special characters. If it finds special characters, it replaces them and then performs an update to the record.
This process is running at a rate of about 6 million records per day, which is too slow for me.
I have tried running the program on in-house fast servers and cloud-virtual servers and the speed is always the same, at about 6 million per 24 hours.
I understand that each record must be checked, and that will take time, but I am wondering if there is any trick to making this run faster, because at the current rate, it will take too long.
If you are an expert with PostgreSQL and are able to provide me with a solution that will make this work faster, then please contact me.
Thank you