Here is a preliminary outline of the module structure and lecture timetable. I will develop most of the ideas on the blackboard. You are encouraged to take notes during the lectures. Any handouts used will be made available here as pdf files shortly after the paper versions have been distributed. Some knowldege of maths is assumed, but more complicated notions will always be explained during the lectures. However, your maths should be better than this.
Week |
Session 1 Mondays 9:00-10:00 |
Session 2 Fridays 13:00-14:00 |
---|---|---|
1 | Module Administration. Introduction to Neural Networks and their History. | Biological Neurons and Neural Networks. Artificial Neurons. [PDF] |
2 | Networks of Artificial Neurons. Single Layer Perceptrons. | Learning and Generalization in Single Layer Perceptrons. |
3 | Perceptron: Training and convergence. [DEMO] | Multi-Layer Perceptrons. |
4 | Feed-forward networks as a function class. | Gradient-based optimization. [PDF] |
5 | Training by Back-Propagation of errors. | Variants and modifications of BackProp. |
6 | Self-Organising Maps. [PDF] (John Bullinaria) | Learning Vector Quantisation. [PDF] (John Bullinaria) |
7 | Neural network applications. [PDF] [PDF] | Statistical perspective on learning. |
8 | Bias-variance dilemma. [PDF] | Dealing with the bias-variance dilemma. |
9 | Ensembles of Neural Networks. [PDF] | Negative correlation in ensembles of neural networks. [PDF] |
10 | Radial basis function networks. | Generalized Radial basis function networks, kernel machines. |
11 | Recurrent neural networks. | Back-Propagation through time, Real-Time recurrent learning. |
12 | Two Revision Lectures Covering the Whole Module |
As you get familiar with different types of neural network models, try them out on benchmark data sets people in the machine learning community have been using to support their claims about yet another excellent learning system :-)
Here are two of the widely used data repisitories that contain data description, data itself and other useful things, like previously obtained results.
Here is a simple example of Generalized Inverse and Correlation matrix memory models.
Take a look at the 8 faces (taken from CMU image data repository within the UCI archive) forming the training data. The last face (lower right corner) is a corrupt version of the first face (upper left corner).
The recall by GI and CMM models
as well as novelty detection from GI can be found
here.
GI is a more complicated model, but gives better results.
What about the new input pattern at the upper left corner of this figure? Anything like it is present in the training corpus, so both models get confused. Note the multiple "ghost" faces in the recall of GI. Novelty detection, however, clearly detects what is wrong with this input.
Data and MATLAB code for producing these pictures are collected in the gzip/tar-ed file. Run gi_cmm_faces.m.
First, implement a feed-forward neural network trained by BackPropagation. Alternatively, make yourself familiar with my implementation in C. Un-tar the file bp.code.tar.gz, go to the folder "BP.CODE" and consult "read.me". There is an example data set in the directory "IONOSPHERE". Preprocessed training inputs and desired outputs (targets) are in files "ionosphere.trn.in" and "ionosphere.trn.nn.t", respectively. Test inputs and targets are in "ionosphere.tst.in" and "ionosphere.tst.nn.t", respectively. Read more about the data set in the "ionosphere.info.txt" file. You can also use one of many neural network simulators available on the web. See section Simulators and Code below.
You can work with the ionosphere data, or choose one of three data sets filed in data.tar.gz. Read the information files (*.info.txt) in the "DATA/IRIS", "DATA/WINE" and "DATA/TIC-TAC-TOE" folders. Alternatively, you can pick any reasonably complex data set of your own choice (e.g. from the web) or making. Please consult me if you decide to go for the last option.
Train and test the networks with
Report the results in a statistically organized manner, i.e. compute simple statistics of performance measures (mean and standard deviation) accross multiple runs of network training/testing. (worth 20%)
Estimate the test error for some of the networks on the original training set using the n-fold cross-validation strategy. Compare with the test error computed on the original test set. Repeat the experiment for n = 5,10,N, where N is the size of the original training set (leave-one-out cross-validation). (worth 20%)
The overall mark m (in the range 0-100%) will be linearly scaled to the range 0-20% by 0.2*m.
Example reports from the past:
- report by Kennon Ballou
- report by Andrew Brown
- report by Michael Nashvili
Kennon, Andrew, Michael - Thank you so much!
For formal details about the aims and objectives and assessment you should look at the official Module Syllabus Page [Neural Comp] [Intro to Neural Comp].
Neural Comp. students will be assessed by examination (100%).
Intro to Neural Comp. students will be assessed based on examination (80%)
and a continuous assessment by mini-project report (20%).
As the material is developed
I will give you
ideas of the standard and type of questions you can expect
in this year's examination.
I will address questions related to the material covered in previous
lectures in great detail during the timetabled
Exercise Sessions.
The Recommended Books for this module are:
Title | Author(s) | Publisher, Date | Comments |
---|---|---|---|
An Introduction to Neural Networks | Kevin Gurney | Routledge, 1997 | Non-mathematical introduction. |
Neural Networks: A Comprehensive Foundation | Simon Haykin | Prentice Hall, 1999 | Very comprehensive, a bit heavy in maths. |
Neural Networks for Pattern Recognition | Christopher Bishop | Clarendon Press, Oxford, 1995 | Highly recommended for mathematically minded students. |
Introduction to Neural Networks | R. Beale & T. Jackson | IOP Publishing, 1990 | Introductory text. |
An Introduction to the Theory of Neural Computation | J. Hertz, A. Krogh & R.G. Palmer | Addison Wesley, 1991 | Good all round book. Slightly mathematical. |
Neural Networks - Notes | I F Wilde, King's College London | Publically available at http://www.mth.kcl.ac.uk/~iwilde/notes/nn/ [PDF] | An excellent set of notes. Highly recommended for mathematically minded students. |
If you have a reasonably mathematical background and you can only afford to buy one book for this module, I would recommend getting the one by Haykin.
If you want to find information about Neural Networks on the web, the best place to start is: The Neural Networks FAQ site which contains a large range of information and links about all aspects of neural networks.
A comprehensive information about resources/activities related to Neural Networks are at: Neural Network Resources.
When programming your own neural networks, it may be useful to consult
the web page created by
John Bullinaria
giving a Step by Step Guide
to Implementing a Simple Neural Network in C.
It should be fairly
straightforward to see how to use it with related programming languages such as C++ and Java.