Class decomposition classification for Big Dataset (Machine Learning)
Budget: $30 – $250 USD
The input will be a data set X with n vectors (points) having d numeric dimensions and one binary class attribute Y .
You will have two choices to reduce d - should NOT USE THE EXISTING LIBRARIES (should code): (1) variable selection, via the correlation matrix, computing correlations between each independent variable and Y . (
2) PCA to reduce d and get representative dimensions in the PCs.
Variable selection: Compute correlation matrix ρ based on Γ. using ρ, select variables highly correlated to Y .
PCA for dimensionality reduction: Compute correlation matrix ρ based on Γ. Compute PCA on ρ, the
correlation matrix of the data set. You will identify a small subset of dimensions that are as independent as possible. You will use PCA taking the top PCs and identifying representative attributes in each PC (stronger correlations to PC). Compare speed with existing PCA in an existing math library.
Classification: the classification models you must program is Class decomposition using Γ (K-means on each class)
You must compare accuracy and speed with some existing classification model. Choose one that you like;
no need to compare against all of them. you can use the existing libraries for this
1. SVM (Gaussian kernel).
2. Discrete Naive Bayes (histograms).
For the classification problem given the input data set, you will need to build a training dataset and a test dataset. You will need to generate a model (classifier/regression) using the training data set. Then, you will apply this model to the test data set and produce a new data set with the predicted class. You will need to generate a Type I, Type II error table (true/false positives, true/false negatives).
i will send the dataset personally (1M). Since it is a large dataset, you should work on chunks
You will have two choices to reduce d - should NOT USE THE EXISTING LIBRARIES (should code): (1) variable selection, via the correlation matrix, computing correlations between each independent variable and Y . (
2) PCA to reduce d and get representative dimensions in the PCs.
Variable selection: Compute correlation matrix ρ based on Γ. using ρ, select variables highly correlated to Y .
PCA for dimensionality reduction: Compute correlation matrix ρ based on Γ. Compute PCA on ρ, the
correlation matrix of the data set. You will identify a small subset of dimensions that are as independent as possible. You will use PCA taking the top PCs and identifying representative attributes in each PC (stronger correlations to PC). Compare speed with existing PCA in an existing math library.
Classification: the classification models you must program is Class decomposition using Γ (K-means on each class)
You must compare accuracy and speed with some existing classification model. Choose one that you like;
no need to compare against all of them. you can use the existing libraries for this
1. SVM (Gaussian kernel).
2. Discrete Naive Bayes (histograms).
For the classification problem given the input data set, you will need to build a training dataset and a test dataset. You will need to generate a model (classifier/regression) using the training data set. Then, you will apply this model to the test data set and produce a new data set with the predicted class. You will need to generate a Type I, Type II error table (true/false positives, true/false negatives).
i will send the dataset personally (1M). Since it is a large dataset, you should work on chunks