I need Pytorch Expert
Budget: £20 – £250 GBP
Set up dataset
You may choose any of the datasets included with Pytorch, or used in the course so far (including, for example, the percolation data). (It is a free country and you may use any other data you wish.)
A safe choice would be either the MNIST data or the MNIST-fashion data, which is a drop-in replacement for MNIST (same size data format, same number of classes, same number of training and test examples).
A more interesting choice would be CIFAR-10
In setting up the data, you should set up a training set and a test set. The test set should be large enough to give a reasonably accurate assessment of the error-rate (or loss) of your models: preferably at least 10,000 examples.
For the learning curve experiment (below), you will need to construct training sets of different sizes, with the largest at least 10 times the smallest. For the MNIST data, for example, your smallest training set might be 500, with sizes 500, 1000, 2000, 4000, 8000, 16000, 32000, and perhaps 60000 if you have time. (You get no extra marks for doing very long experiments.)
Set up model
Set up a Pytorch model (you may find it helpful to keep it simple and fast to train). Train on an intermediate-sized dataset (if you are using MNIST, say 2000 or 4000).
Plot the loss and error rate as a function of training epochs.
Ensure that your model is complex enough to overfit the training data: that is the loss/error-rate on the training set should be below your target loss/error-rate, and the loss/error-rate on the validation set should be higher than this (preferably higher than your target error-rate).
2. Regularisation experiment:
Now select a type of regularisation (which may be L2, or dropout) and train your initial model with different amounts of regularisation.
A small amount of regularisation may improve the validation set performance of your overfitted model; too much regularisation may make performance on the validation set worse.
Train your model multiple times, applying different amounts of regularisation, and plot a graoph showing the effect of different amounts of regularisation.
Plot the amount of regularisation along the x axis, and the validation set performance (loss/error-rate) on the y axis. How does regularisation affect the performance of your initial network, and what is the optimal amount?
3. Learning curve experiment:
Now train your model (with the amount of regularisation you selected) on different amounts of training data.
Plot the performance (loss/error-rate) of the model on the validation set against the size of the training set. This is typically best done on a log-log plot.
Describe the approximate relationship between the training set size and loss / error-rate. Does the network performance appear to improve as some power of the amount of data in the training set? If so, by what power ?
For example, a very good rate of improvement is for error-rate to be proportional to 1?⎯⎯√
where ? is the training set size. For your model, the rate of improvement of validation error with training set size may not be as fast as this (or it may, who knows?) The aim of this exercise is to find out what it actually is.
4. Second model:
Devise what you believe to be a better neural network architecture for the problem (e.g. for MNIST you might use a fully-connected network for your initial model, and then try a convolutional net for your second model).
Repeat sections 2 (regularisation experiment), and determine whether it has better validation set performance than the first model, for an intermediate size of training set (e.g. 2000 or 4000 for the MNIST data).
Repeat section 3. Does your model have a different learning curve from the first? Plot the learning curves for the first and second model on the same graph, to compare them. Comment: is there a more rapid reduction of error-rate with training set size for your second model?
You may choose any of the datasets included with Pytorch, or used in the course so far (including, for example, the percolation data). (It is a free country and you may use any other data you wish.)
A safe choice would be either the MNIST data or the MNIST-fashion data, which is a drop-in replacement for MNIST (same size data format, same number of classes, same number of training and test examples).
A more interesting choice would be CIFAR-10
In setting up the data, you should set up a training set and a test set. The test set should be large enough to give a reasonably accurate assessment of the error-rate (or loss) of your models: preferably at least 10,000 examples.
For the learning curve experiment (below), you will need to construct training sets of different sizes, with the largest at least 10 times the smallest. For the MNIST data, for example, your smallest training set might be 500, with sizes 500, 1000, 2000, 4000, 8000, 16000, 32000, and perhaps 60000 if you have time. (You get no extra marks for doing very long experiments.)
Set up model
Set up a Pytorch model (you may find it helpful to keep it simple and fast to train). Train on an intermediate-sized dataset (if you are using MNIST, say 2000 or 4000).
Plot the loss and error rate as a function of training epochs.
Ensure that your model is complex enough to overfit the training data: that is the loss/error-rate on the training set should be below your target loss/error-rate, and the loss/error-rate on the validation set should be higher than this (preferably higher than your target error-rate).
2. Regularisation experiment:
Now select a type of regularisation (which may be L2, or dropout) and train your initial model with different amounts of regularisation.
A small amount of regularisation may improve the validation set performance of your overfitted model; too much regularisation may make performance on the validation set worse.
Train your model multiple times, applying different amounts of regularisation, and plot a graoph showing the effect of different amounts of regularisation.
Plot the amount of regularisation along the x axis, and the validation set performance (loss/error-rate) on the y axis. How does regularisation affect the performance of your initial network, and what is the optimal amount?
3. Learning curve experiment:
Now train your model (with the amount of regularisation you selected) on different amounts of training data.
Plot the performance (loss/error-rate) of the model on the validation set against the size of the training set. This is typically best done on a log-log plot.
Describe the approximate relationship between the training set size and loss / error-rate. Does the network performance appear to improve as some power of the amount of data in the training set? If so, by what power ?
For example, a very good rate of improvement is for error-rate to be proportional to 1?⎯⎯√
where ? is the training set size. For your model, the rate of improvement of validation error with training set size may not be as fast as this (or it may, who knows?) The aim of this exercise is to find out what it actually is.
4. Second model:
Devise what you believe to be a better neural network architecture for the problem (e.g. for MNIST you might use a fully-connected network for your initial model, and then try a convolutional net for your second model).
Repeat sections 2 (regularisation experiment), and determine whether it has better validation set performance than the first model, for an intermediate size of training set (e.g. 2000 or 4000 for the MNIST data).
Repeat section 3. Does your model have a different learning curve from the first? Plot the learning curves for the first and second model on the same graph, to compare them. Comment: is there a more rapid reduction of error-rate with training set size for your second model?
Related categories:
Deep Learning