Apache Spark machine learning project

There are three different problems that you must solve. The first two problems the
datasets and problem statement are given to you. The last problem is open ended
where you must obtain a machine learning challenge online and solve it.
1. Problem 1
The “diabetes.csv” dataset has historical data on individuals that eventually either
developed diabetes or not. Diabetes is a condition where the body does not produce
enough insulin to break down the food that you eat. Without medication, diabetes
can lead to damage to cells and vital organs and eventual death.
In this problem, we want to predict whether a person is at risk of becoming diabetic
based on the individual’s data. This information can then be used to begin
preventative measures for the individual (example lifestyle change in diet and
exercise). The features and label for the dataset are described below
Features or independent Variables:
Pregnancies: Number of times pregnant
Glucose: Plasma glucose concentration, 2 hours in an oral glucose tolerance test
BloodPressure: Diastolic blood pressure (mm Hg)
SkinThickness: Triceps skin fold thickness (mm)
Insulin: 2-Hour serum insulin (mu U/ml)
BMI: Body mass index (weight in kg/(height in m)^2)
DiabetesPedigreeFunction: Diabetes pedigree function, a score based on genetic
factor of a person (diabetes has a close relation to family history).
Age: Age (years)
Labels or Dependent Variable:
Outcome: No Diabetes = 0, Diabetes=1
2. Problem 2
The “cancer.csv” dataset deals with cancer patients. A tumour is a set of cells that
have grown in a specific part of body. Tumours can be classified as being either
cancerous or non-cancerous based on various factors. Cancerous tumours continue
to grow uncontrollably and spread to different parts of the body and eventually to
the bloodstream. At this stage, they begin interfering with body functions that can
lead to death (example heart attack from clogged arteries). The reason it is important
to classify tumours correctly is because generally it is expensive and risky to try to
remove all tumours. In this problem, we want to predict whether a person’s tumour
is cancerous in order to decide whether surgery is necessary or not.
Features or Independent Variables:
ID - Sample code number
Clump Thickness: 1 - 10
Uniformity of Cell Size: 1 - 10
Uniformity of Cell Shape: 1 - 10
Marginal Adhesion: 1 - 10
Single Epithelial Cell Size: 1 - 10
Bare Nuclei: 1 - 10
Bland Chromatin: 1 - 10
Normal Nucleoli: 1 - 10
Mitoses: 1 - 10
Label or Dependent Variable:
Class: (2 for benign, 4 for malignant)
3. Problem 3
Kaggle.com is a website that host datasets and machine learning challenges. Explore
the machine learning challenges from this site
https://www.kaggle.com/competitions
Choose one problem and solve it using SparkML. The problem (and hence the alyou
chose) should be something other than a classification problem.
Tasks:
For the following tasks, make sure that all your variable names have your name at
the end to identify that you have done the work, for example:
val data_saber = spark.read .format("csv") .option("header", "true")
.load("hdfs://10.128.0.55:8020/BigData/cancer.csv")
2. For each algorithm, make sure to have each of the following
• Use a training and test set for evaluation
• Use at least 2 hyperparameters and for each hyperparameter give at least
three (3) values. Furthermore, divide the dataset to 3 .
Make sure to describe what the hyperparameters are
• For all algorithms, make sure to use an appropriate evaluation criteria. You
may need to research evaluation criteria for the algorithm you choose.
3. Write a report highlighting each model’s results including discussion of the
hyperparameters used
Deliverables:
1. Provide the code for each algorithm (3 files) in a plain text file.
2. Provide a report in Word or PDF format for the report portion (1 file)
In total you should have 4 files. Do not compress these files (convert to zip or rar or
other compressed formats).