This module is assessed by 80% examination and 20% continuous assessment. This document specifies the continuous assessment component.

The objective of this exercise is for you to gain practical experience in designing, implementing, training and optimizing a neural network designed to perform a real world task.

This year the task is Optical Recognition of Handwritten Digits, and the data sets you must use are provided by the UCI Machine Learning Repository.

The paper copy of your report and the accompanying CD/DVD must be handed in to the School Office by 12noon on Wednesday 11 January 2012. If you miss this deadline, your mark will be reduced by 5% (out of 100%) for each School working day (or part thereof) your hand-in is late.

Feedback with marks will be returned within four weeks of the hand-in deadline.

What You Need To Do

(1) Download the data sets and descriptions.

(2) Design a suitable Multiple-Layer Perceptron neural network that can learn from the training data set to generalize well to the testing data set. Hint: Some form of Back-Propagation learning algorithm will probably be most appropriate.

(3) Implement your neural network, using any programming language you like, or by setting it up in an existing neural network simulator. You may use any code or simulator you find on the internet, as long as you properly reference the sources in your report, or you can program your own neural network from scratch. If, like most students have done in the past, you decide to write your own code, you may find John Bullinaria's Step by Step Guide to Implementing a Simple Neural Network in C helpful to get you started. Hints: Writing your own code usually offers more flexibility than using someone else's simulator, though it may prove more time consuming, and there is limited credit/marks available for the implementation aspect of this exercise.

(4) Experiment systematically with the things that you think are most likely to improve the generalization performance you achieve. Hints: Consider how you are going to avoid under-fitting and over-fitting of the training data. Don't get carried away - aim to test two or three things well, rather than large numbers of things not very well.

(5) Write a report explaining what you did and what you found. You should include the following sections:

1. Introduction - Say what the data sets involved and what you aimed to achieve. [5%]

2. Design - Describe and justify the neural network you designed for the task, and the factors you decided to experiment with. [15%]

3. Implementation - Describe how you implemented your network and associated analysis mechanisms. Explain why you chose to do it that way. Remember to cite any sources you used. [20%]

4. Experiments - Describe the experiments you performed and the results you obtained. Explain exactly how you used the training and testing data sets. The results should be presented in a statistically organized manner, e.g. by computing simple statistics of performance measures (such as means and standard deviations of percentages correct) across multiple runs of network training/testing, and plotting graphs with error-bars. [50%]

5. Conclusions - Comment on whether the things you varied had any effect, and describe the best generalization result you achieved. [10%]

Don't forget that this exercise corresponds to only 20% of a 10 credit module. Spend an appropriate amount of time on it!

Assessment

You need to submit the written report as described above, plus a CD/DVD containing a .pdf version of your report, source files for your report (e.g., LaTeX or MS Word files), the source code of your neural network with instructions for using it, and any results files you think appropriate.

A reasonable length for the report would be between 2000 and 4000 words, plus as many diagrams, tables and graphs as you think appropriate. Marks will be awarded according to the percentage proportions indicated above, based on what you did and how well you described it.

Help and Advice

There will be an experienced Research Fellow available to advise you on all aspects of this exercise - Chris Bowers. He will be available for the rest of this term during his office hour on Thursdays at 10:00am in Room 223 of the School of Computer Science.