Image Segmentation Program Making
Budget: $10 – $30 USD
In this project, we will test machine learning models for image segmentation task. The problem of
image segmentation can be formulated as a classification task, where every pixel in an image is
mapped to an already known class C based on the information extracted from its N x N neighbors.
The following figures illustrate an input example for the image segmentation task and its
corresponding segmentation labels.
A set of images and their ground-truth segmentation maps is provided to train and evaluate your
models. An image is in principle a three-dimensional array of size H x W x 3, where H is the image
height, W is the image width, and 3 corresponds to the three RGB channels. Each location in the 2d
grid is called a pixel and each of its RGB channels has a value between 0 and 255 that represent the
intensity of the color. For example, a pixel with value [255, 0, 0] corresponds to a red pixel, whereas
a grey pixel will have a value [127, 127, 127].
For our application, each pixel in an image is labeled with a unique class C in [Grass = 3, Water = 2,
Cow = 1, Others = 0] as specified in its segmentation map. Separate train_images.txt and
test_images.txt files are provided to specify training and testing images respectively.
For all tested model, the input will be an image patch of size 17 x 17 and the target will be the class
label for the center pixel of the patch.
You need to complete the following tasks:
1. Training Data Generation
Create a balanced training set by randomly sampling image patches of size 17 x 17 from the training
images and assigning the class label of the center pixel as a target for the image patch. Sample at
Input image Ground-truth segmentation map
least 5000 patches for each class (i.e., you will have a training set that contains at least 20000
examples.)
Create another balanced validation set by sampling around 800 image patches per class from the
training images. Note that the validation set and the training set are disjoint.
For the test set, we will use all the image patches in the testing images.
2. Features extraction
We will represent each image patch with a 192-dimensional features vector that corresponds to the
intensity histograms for each of the RGB channels in the patch.
For each of the RGB channels, create a histogram with 64 bins, where bin i contains the number of
pixels with intensity value in the range [i*4, (i+1)*4), and 0 i 64.
Normalize the histogram of each channel by dividing by the number of pixels in the patch.
The final features vector is obtained by concatenating the normalized histogram for all the channels.
3. Baseline Model
As a baseline model, evaluate a nearest neighbor baseline (with k=1). Report the pixel-wise accuracy
on the test images.
4. Improvements
Try to achieve better performance by evaluating two additional models on the task. Discuss and
motivate your model selection, and comment on why the performance has improved (or potentially
did not improve). If the chosen models have hyper-parameters, make sure to tune at least one
hyper-parameter for each model using the validation set. To tune a parameter, you need to test at
least three different values.
For each model report the pixel-wise accuracy on the training, validation, and testing sets.
Visualize the predictions for both models on the test images (same as the figure above). Examine
the visualized predictions and check the wrongly predicted labels. Discuss any interesting
observation you notice on the prediction errors.
image segmentation can be formulated as a classification task, where every pixel in an image is
mapped to an already known class C based on the information extracted from its N x N neighbors.
The following figures illustrate an input example for the image segmentation task and its
corresponding segmentation labels.
A set of images and their ground-truth segmentation maps is provided to train and evaluate your
models. An image is in principle a three-dimensional array of size H x W x 3, where H is the image
height, W is the image width, and 3 corresponds to the three RGB channels. Each location in the 2d
grid is called a pixel and each of its RGB channels has a value between 0 and 255 that represent the
intensity of the color. For example, a pixel with value [255, 0, 0] corresponds to a red pixel, whereas
a grey pixel will have a value [127, 127, 127].
For our application, each pixel in an image is labeled with a unique class C in [Grass = 3, Water = 2,
Cow = 1, Others = 0] as specified in its segmentation map. Separate train_images.txt and
test_images.txt files are provided to specify training and testing images respectively.
For all tested model, the input will be an image patch of size 17 x 17 and the target will be the class
label for the center pixel of the patch.
You need to complete the following tasks:
1. Training Data Generation
Create a balanced training set by randomly sampling image patches of size 17 x 17 from the training
images and assigning the class label of the center pixel as a target for the image patch. Sample at
Input image Ground-truth segmentation map
least 5000 patches for each class (i.e., you will have a training set that contains at least 20000
examples.)
Create another balanced validation set by sampling around 800 image patches per class from the
training images. Note that the validation set and the training set are disjoint.
For the test set, we will use all the image patches in the testing images.
2. Features extraction
We will represent each image patch with a 192-dimensional features vector that corresponds to the
intensity histograms for each of the RGB channels in the patch.
For each of the RGB channels, create a histogram with 64 bins, where bin i contains the number of
pixels with intensity value in the range [i*4, (i+1)*4), and 0 i 64.
Normalize the histogram of each channel by dividing by the number of pixels in the patch.
The final features vector is obtained by concatenating the normalized histogram for all the channels.
3. Baseline Model
As a baseline model, evaluate a nearest neighbor baseline (with k=1). Report the pixel-wise accuracy
on the test images.
4. Improvements
Try to achieve better performance by evaluating two additional models on the task. Discuss and
motivate your model selection, and comment on why the performance has improved (or potentially
did not improve). If the chosen models have hyper-parameters, make sure to tune at least one
hyper-parameter for each model using the validation set. To tune a parameter, you need to test at
least three different values.
For each model report the pixel-wise accuracy on the training, validation, and testing sets.
Visualize the predictions for both models on the test images (same as the figure above). Examine
the visualized predictions and check the wrongly predicted labels. Discuss any interesting
observation you notice on the prediction errors.