Help with loading huge MongoDB data set
Budget: $30 – $250 USD
We are looking for a MongoDB expert who has experience with very large databases.
We want to port a database from an existing SQL environment to MongoDB.
The database has about 1.5 billion records.
PART 1:
The records are large records and there are several indexes.
We need someone who has experience with loading this much data into a single MongoDB collection (table).
We have already attempted to do this and although we have managed to get the data into MongoDB is was a very painful process which we can not reproduce.
Our understanding is that MongoDB was made for very large sets of data, so we cannot understand why it was so difficult.
We now need someone to make a step by step guide for us explaining how and what to do to load the data, and build the indexes.
You MUST be a MongoDB expert with first hand experience manipulating very large data sets.
PART 2:
The data in the database includes a field called TRANSACTION_TYPE and another field called TRANSACTION_SUB_TYPE.
We have an index on TRANSACTION_TYPE and TRANSACTION_SUB_TYPE.
We want to be able to display the list of all transactions for a specific type.
If the total list has more than 100 records (which is most of the time), then we want 100 records to be listed at a time, so we have created a page field called TRANSACTION_TYPE_PAGE.
In order for us to be able to display like this we created an index on TRANSACTION_TYPE, TRANSACTION_SUB_TYPE and TRANSACTION_TYPE_PAGE.
So to prepare the data and make this work, we need to loop through all 1.5 billion records, using the index on TRANSACTION_TYPE and TRANSACTION_SUB_TYPE and count them as we are looping.
As we loop we set the TRANSACTION_TYPE_PAGE, and then every time we reach 100 we set the page to page+1.
So if we have 7 million transactions for type A, they will be displayed over 70,000 pages, 100 records per page.
They will also be sorted by TRANSACTION_SUB_TYPE, due to the index, so type A, sub_type A (AA), AB, AC etc.
And when we see a new TRANSACTION_TYPE the page gets reset back to 1, and we start again.
In this way when we search for TRANSACTION_TYPE="PURCHASE" and it returns 7,000,000 records, we want to be able to display them over 70,000 pages at 100 per page, and sorted by TRANSACTION_SUB_TYPE.
So to make this work faster, we created an index on TRANSACTION_TYPE and TRANSACTION_SUB_TYPE and TRANSACTION_TYPE_PAGE.
Therefore if we query the database and ask for transaction type A, page 200, we should immediately get 100 records.
So we need someone who can explain to us how to set the page numbers in this way.
The current MongoDB script that we used was generated by ChatGPT but it takes too long to number all the pages of the 1.5 billion records.
As we will be adding about 50 million transactions a month, we will need to renumber the entire database again every time we add new records.
So we need someone to create a program that can do this as fast as possible.
===
We currently use php programs to get the data from MongoDB and display it and it works fine.
You MUST have REAL MongoDB experience and experience with huge data sets. Please do not bid on this project if you have never done work like this before.
Please read and understand the project details carefully. Your bid on this project is your final bid. If you are awarded the project you cannot ask for more money or a tip after the project is awarded. You will be paid what you bid. If you have any questions, please ask them before you bid. Your level of professionalism will determine if I do future work with you as there are more parts to this project to come after this one.
Thank you
We want to port a database from an existing SQL environment to MongoDB.
The database has about 1.5 billion records.
PART 1:
The records are large records and there are several indexes.
We need someone who has experience with loading this much data into a single MongoDB collection (table).
We have already attempted to do this and although we have managed to get the data into MongoDB is was a very painful process which we can not reproduce.
Our understanding is that MongoDB was made for very large sets of data, so we cannot understand why it was so difficult.
We now need someone to make a step by step guide for us explaining how and what to do to load the data, and build the indexes.
You MUST be a MongoDB expert with first hand experience manipulating very large data sets.
PART 2:
The data in the database includes a field called TRANSACTION_TYPE and another field called TRANSACTION_SUB_TYPE.
We have an index on TRANSACTION_TYPE and TRANSACTION_SUB_TYPE.
We want to be able to display the list of all transactions for a specific type.
If the total list has more than 100 records (which is most of the time), then we want 100 records to be listed at a time, so we have created a page field called TRANSACTION_TYPE_PAGE.
In order for us to be able to display like this we created an index on TRANSACTION_TYPE, TRANSACTION_SUB_TYPE and TRANSACTION_TYPE_PAGE.
So to prepare the data and make this work, we need to loop through all 1.5 billion records, using the index on TRANSACTION_TYPE and TRANSACTION_SUB_TYPE and count them as we are looping.
As we loop we set the TRANSACTION_TYPE_PAGE, and then every time we reach 100 we set the page to page+1.
So if we have 7 million transactions for type A, they will be displayed over 70,000 pages, 100 records per page.
They will also be sorted by TRANSACTION_SUB_TYPE, due to the index, so type A, sub_type A (AA), AB, AC etc.
And when we see a new TRANSACTION_TYPE the page gets reset back to 1, and we start again.
In this way when we search for TRANSACTION_TYPE="PURCHASE" and it returns 7,000,000 records, we want to be able to display them over 70,000 pages at 100 per page, and sorted by TRANSACTION_SUB_TYPE.
So to make this work faster, we created an index on TRANSACTION_TYPE and TRANSACTION_SUB_TYPE and TRANSACTION_TYPE_PAGE.
Therefore if we query the database and ask for transaction type A, page 200, we should immediately get 100 records.
So we need someone who can explain to us how to set the page numbers in this way.
The current MongoDB script that we used was generated by ChatGPT but it takes too long to number all the pages of the 1.5 billion records.
As we will be adding about 50 million transactions a month, we will need to renumber the entire database again every time we add new records.
So we need someone to create a program that can do this as fast as possible.
===
We currently use php programs to get the data from MongoDB and display it and it works fine.
You MUST have REAL MongoDB experience and experience with huge data sets. Please do not bid on this project if you have never done work like this before.
Please read and understand the project details carefully. Your bid on this project is your final bid. If you are awarded the project you cannot ask for more money or a tip after the project is awarded. You will be paid what you bid. If you have any questions, please ask them before you bid. Your level of professionalism will determine if I do future work with you as there are more parts to this project to come after this one.
Thank you