Data Analytics in Big Data
Budget: $30 – $250 USD
Solve the imbalanced classification problem associated with the ECBDL14 problem, with a sub-problem that has 960,000 instances, 90 attributes, and the following class distribution:
• Class 1.0: 19,089 instances.
• Class 0.0: 940,911 instances.
For the test set, 240,000 instances will be used with the following distribution:
• Class 1.0: 4,911 instances.
• Class 0.0: 235,089 instances.
The problem must be solved using the MLLib library and algorithms available in Spark Packages. You must use at least 4 learning algorithms and 4 preprocessing algorithms:
a) Learning: From the MLlib library, use the Decision Tree, Random Forest, and another of the student's choice. Additionally, use at least one other algorithm from the Spark Package repository.
b) Preprocessing: Use data balancing preprocessing algorithms ROS and RUS1, and at least two additional preprocessing algorithms.
It is recommended to use some of the preprocessing algorithms discussed in class.
You must describe in detail in a PDF file the entire algorithmic process used, showing the results of each of the algorithms used for training and testing, analyzing the behavior of the algorithms, and showing the flows/combinations of preprocessing algorithms. An analysis of data redundancy associated with the practice dataset must be performed.
Additionally, you must attach the scripts (this practice must be done in Scala)
Performance Measure
The performance evaluation metric used is the TPR x TNR on the test set (the product of the classification rates for each class). The objective is to maximize this performance measure in the study to be conducted. Please remember:
Execution instructions on the cluster:
• Do NOT use spark-shell. Use spark-submit.
• Limit the number of nodes to 14.
• Limit the memory used to 4 GB.
Data set:
• Header: /user/datasets/ecbdl14/ECBDL14-1.2mill-90.header
• Train: /user/datasets/ecbdl14/ECBDL14-1.2mill-90-5-1tra.dat
• Test: /user/datasets/ecbdl14/ECBDL14-1.2mill-90-5-1tst.dat
• Class 1.0: 19,089 instances.
• Class 0.0: 940,911 instances.
For the test set, 240,000 instances will be used with the following distribution:
• Class 1.0: 4,911 instances.
• Class 0.0: 235,089 instances.
The problem must be solved using the MLLib library and algorithms available in Spark Packages. You must use at least 4 learning algorithms and 4 preprocessing algorithms:
a) Learning: From the MLlib library, use the Decision Tree, Random Forest, and another of the student's choice. Additionally, use at least one other algorithm from the Spark Package repository.
b) Preprocessing: Use data balancing preprocessing algorithms ROS and RUS1, and at least two additional preprocessing algorithms.
It is recommended to use some of the preprocessing algorithms discussed in class.
You must describe in detail in a PDF file the entire algorithmic process used, showing the results of each of the algorithms used for training and testing, analyzing the behavior of the algorithms, and showing the flows/combinations of preprocessing algorithms. An analysis of data redundancy associated with the practice dataset must be performed.
Additionally, you must attach the scripts (this practice must be done in Scala)
Performance Measure
The performance evaluation metric used is the TPR x TNR on the test set (the product of the classification rates for each class). The objective is to maximize this performance measure in the study to be conducted. Please remember:
Execution instructions on the cluster:
• Do NOT use spark-shell. Use spark-submit.
• Limit the number of nodes to 14.
• Limit the memory used to 4 GB.
Data set:
• Header: /user/datasets/ecbdl14/ECBDL14-1.2mill-90.header
• Train: /user/datasets/ecbdl14/ECBDL14-1.2mill-90-5-1tra.dat
• Test: /user/datasets/ecbdl14/ECBDL14-1.2mill-90-5-1tst.dat