In this tutorial you will learn about:
- Unsupervised Learning
- Kohonen Networks
- Learning in Kohonen Networks
In all the forms of learning we have met so far the answer that the
network is supposed to give for the training examples is known. That
type of learning requires a teacher who knows the correct
classification for the input patterns in the training set. The
objective is typically to generalise from these to other, previously
unseen examples: giving more or less correct answers without
In unsupervised learning the aim is rather different. The objective
is, put most simply, to find the natural structure inherent in the
input data. There are a number of unsupervised learning schemes,
including competitive learning, adaptive resonance theory and
Self-Organising Feature Maps (SOFMs). A well known type of SOFM is a
The objective of a Kohonen network is to map input vectors (patterns)
of arbitrary dimension N onto a discrete map with 1 or 2
dimensions. Patterns close to one another in the input space should be
close to one another in the map: they should be topologically ordered.
A Kohonen network is composed of a grid of output units and N input
units. The input pattern is fed to each output unit. The input lines
to each output unit are weighted. These weights are initialised to
small random numbers.
Learning in Kohonen Networks
The learning process is as roughly as follows:
The winning output unit is simply the unit with the weight vector that
has the smallest Euclidean distance to the input pattern. The
neighbourhood of a unit is defined as all units within some distance
of that unit on the map (not in weight space). In the demonstration
below all the neighbourhoods are square. If the size of the
neighbourhood is 1 then all units no more than 1 either horizontally
or vertically from any unit fall within its neighbourhood.
The weights of every unit in the neighbourhood of the winning unit
(including the winning unit itself) are updated using
- initialise the weights for each output unit
- loop until weight changes are negligible
- for each input pattern
- present the input pattern
- find the winning output unit
- find all units in the neighbourhood of the winner
- update the weight vectors for all those units
- reduce the size of neighbourboods if required
This will move each unit in the neighbourhood closer to the input
pattern. As time progresses the learning rate and the neighbourhood
size are reduced. If the parameters are well chosen the final network
should capture the natural clusters in the input data.
- Place several clusters of a few points each in the input space
and train the network, reducing the neighbourhood size and the
learning rate gradually, until the changes are negligible. Record the
behaviour of the weight vectors. Re-randomise the weights and repeat
- Why do the weight vectors cluster together to begin with?
- What happens as the neighbourhood size is reduced?
- What happens as the learning rate is reduced?
- Describe the final maps obtained. What is the relationship
between the input data and the map. How well does the map summarise
the input data?
- To demonstrate the networks ability for data compression reduce
the size of the network to 25 output units and train it on a problem
with up to 100 input patterns. Record your observations and explain
why the network behaves in the way it does.
- Now try a problem where the number of output units is far more
than the number of input patterns. What is noticeable about the final
state of the network? Why has this happened?
- Now try presenting problems where the data are grouped in more
complex ways (e.g. an ellipse with a cluster in the centre). Why does
the Kohonen network not summarise the natural regularities in some of
these input sets as well as others?
- The application of Kohonen networks is typically in mapping very
high dimensional input spaces into 1 or 2 dimensions. Devise a problem
with high dimensional inputs, write code to implement your own Kohonen
network and conduct appropriate experiments.