Preparing and uploading datasets from multiple files for machine learning
Budget: $30 – $250 USD
I am currently building a gesture detection system using a random forest machine learning technique. I have 6 different gestures (clockwise, swipe, open fist etc). I have collected data for each gesture from 2 participants currently. Although, I am struggling to get my head around uploading and organizing the data for my project ready for training, testing, and validation.
In my past projects, all my data has been contained within one csv file and uploaded using pandas. Every row within this csv file being a series of features with the last variable being it's resulting target. Where as this project is a lot different to me, there are far more folders and files to work with and I am struggling to find the answer online. This is how the data i have collected is set out:
I currently have two folders, participant 1, participant 2.
Within those folders are 6 more folders to represent each gesture
(clockwise, swipe, open fist etc)
Within each and every gesture folder are 30 csv files which are 30
repetitions to that specific gesture
The csv files then contain two columns, Time and Voltage with 15,007 data points. The target is classified by the folder it is stored under
For example - Participant1/Clockwise/Clockwise_23.csv
What is the most effective way of loading every participants data ready for learning, testing and validating. Adding a target to every csv file by the folder it is stored. And also combining the data collected from both participants under the same gesture. For an example - having an array of all the anticlockwise files from every participants (30 reps x Participant 1 AND 30 reps x Participant 2)
With all these files being loaded - will this be time effective - do I want to be storing a ginormous number of files to memory?
In my past projects, all my data has been contained within one csv file and uploaded using pandas. Every row within this csv file being a series of features with the last variable being it's resulting target. Where as this project is a lot different to me, there are far more folders and files to work with and I am struggling to find the answer online. This is how the data i have collected is set out:
I currently have two folders, participant 1, participant 2.
Within those folders are 6 more folders to represent each gesture
(clockwise, swipe, open fist etc)
Within each and every gesture folder are 30 csv files which are 30
repetitions to that specific gesture
The csv files then contain two columns, Time and Voltage with 15,007 data points. The target is classified by the folder it is stored under
For example - Participant1/Clockwise/Clockwise_23.csv
What is the most effective way of loading every participants data ready for learning, testing and validating. Adding a target to every csv file by the folder it is stored. And also combining the data collected from both participants under the same gesture. For an example - having an array of all the anticlockwise files from every participants (30 reps x Participant 1 AND 30 reps x Participant 2)
With all these files being loaded - will this be time effective - do I want to be storing a ginormous number of files to memory?