Deep Learning

design and train a classifier and a generative model using the CIFAR-100 dataset.
While this might seem straightforward, the challenge is that you are only allowed to train small models
(with a limited number of total parameters) for a short amount of training time (gradient update steps).
You are also to write a short scientific report detailing both methods, experimental results, and limitations
in a provided LATEX template that closely follows parts of the ICLR conference style guidelines. These files
must be zipped together like this, replacing the username with your CIS username. You may not submit
additional source code files:
username.zip
username-paper.pdf
username-generative-model.ipynb (or .py)
username-classifer.ipynb (or .py)
To assist in this, the following template reports and starter code are provided to build on:
W [Deep Learning Paper Template] - login with Durham email, on ‘overleaf.com’ click ‘make a copy’ to edit
W [Google Colab Discriminative Model Starter Code]
W [Google Colab Generative Model Starter Code]
The deep discriminative model
Design and train a CIFAR-100 classifier, reporting both the training and testing accuracy as in the above
starter code. However, your model must have fewer than 100,000 (one hundred thousand) parameters
(this is a low number of parameters for such a task). To display the total number of parameters for a deep
neural network N, you can use something similar to the following code:
print(f'> Number of parameters {len(torch.nn.utils.parameters to vector(N.parameters()))}')
Furthermore, you may not train the neural network for more than 10,000 (ten thousand) optimisation
steps. You must clearly state the total number of parameters and optimisation steps in both the code
and report.
Exceeding either of these constraints will result in a 5 mark penalty for every 10% exceeded. For example,
a model of 110,000 parameters incurs -5 marks whereas 150,000 parameters incurs -25 marks. A model
trained for 11,000 steps incurs -5 marks and training for 15,000 steps incurs -25 marks.
You should design the architecture yourself based on content covered in the lectures, practicals, and
supplementary reading. If you reference existing code, this must be cited clearly in both the submitted
python code and in the .pdf report.
As in the guidance paper template, your report should include: (i) a plot of the training and test accuracy
over the length of your training, (ii) the total number of parameters in your network, (iii) the final values
for training loss, training accuracy and test accuracy (means and standard deviations) as in the format
provided to you in the discriminative model starter code.
You will be assessed primarily based on the quality of the report and the model accuracy. Further details
of how this is graded are given in the marking scheme.
The deep generative model
Using the CIFAR-100 dataset, train a deep generative model to synthesise unique images that will be
judged on their realism, diversity and uniqueness from the original training data.
Limitations in terms of model size and training length also apply to the generative modelling task. Your
model must have fewer than 1,000,000 (one million) parameters. Note that if your approach consists of
multiple networks (e.g., like a GAN), the parameter limit applies to the whole model with all the networks
combined (e.g., the parameter count of the generator and the discriminator added together should be
less that 1,000,000).
Furthermore, you may not train the neural network for more than 50,000 (fifty thousand) optimisation
steps. Each optimisation step is counted when your entire model is trained by calculating the gradients
for the whole model once.
You must clearly state the total number of parameters and optimisation steps for the generative model
of your choice in both the code and report.
Exceeding either of these constraints will result in a 5 mark penalty for every 10% exceeded, for example,
a model of 1,100,000 parameters incurs -5 marks whereas 1,500,000 parameters incurs -25 marks. A
generative model trained for 55,000 steps incurs -5 marks and training for 75,000 steps incurs -25 marks.
In the report, you must display (i) a unique batch of 64 non cherry-picked model samples, (ii) interpolations
between 8 pairs of your samples, and (iii) you must provide FID scores between 10k model samples and
the 10k images in the CIFAR-100 test dataset. You are not permitted to train on the test data.
You can use CIFAR-100 class labels to aid your training and sampling without penalty. You are permitted
to train on a subset of the CIFAR-100 dataset, but you will not score as highly in diversity by doing so. Your
model should generate samples based on a noise vector z, drawn from a prior distribution, rather than
being conditioned on x. In other words, the samples should not be derived from a function of x during
inference.