Kohonen Networks

Introduction

In this tutorial you will learn about:

Unsupervised Learning

In all the forms of learning we have met so far the answer that the network is supposed to give for the training examples is known. That type of learning requires a teacher who knows the correct classification for the input patterns in the training set. The objective is typically to generalise from these to other, previously unseen examples: giving more or less correct answers without intervention. In unsupervised learning the aim is rather different. The objective is, put most simply, to find the natural structure inherent in the input data. There are a number of unsupervised learning schemes, including competitive learning, adaptive resonance theory and Self-Organising Feature Maps (SOFMs). A well known type of SOFM is a Kohonen network.

Kohonen Networks

The objective of a Kohonen network is to map input vectors (patterns) of arbitrary dimension N onto a discrete map with 1 or 2 dimensions. Patterns close to one another in the input space should be close to one another in the map: they should be topologically ordered. A Kohonen network is composed of a grid of output units and N input units. The input pattern is fed to each output unit. The input lines to each output unit are weighted. These weights are initialised to small random numbers.

Learning in Kohonen Networks

The learning process is as roughly as follows: The winning output unit is simply the unit with the weight vector that has the smallest Euclidean distance to the input pattern. The neighbourhood of a unit is defined as all units within some distance of that unit on the map (not in weight space). In the demonstration below all the neighbourhoods are square. If the size of the neighbourhood is 1 then all units no more than 1 either horizontally or vertically from any unit fall within its neighbourhood. The weights of every unit in the neighbourhood of the winning unit (including the winning unit itself) are updated using

\begin{displaymath}\vec{w}_{i} = \vec{w}_{i} + \alpha \, ( \vec{x}_{i} - \ vec{w}_{i}) \end{displaymath} (21)

This will move each unit in the neighbourhood closer to the input pattern. As time progresses the learning rate and the neighbourhood size are reduced. If the parameters are well chosen the final network should capture the natural clusters in the input data.

Demonstration

Exercises

  1. Place several clusters of a few points each in the input space and train the network, reducing the neighbourhood size and the learning rate gradually, until the changes are negligible. Record the behaviour of the weight vectors. Re-randomise the weights and repeat this process.

  2. Why do the weight vectors cluster together to begin with?

  3. What happens as the neighbourhood size is reduced?

  4. What happens as the learning rate is reduced?

  5. Describe the final maps obtained. What is the relationship between the input data and the map. How well does the map summarise the input data?

  6. To demonstrate the networks ability for data compression reduce the size of the network to 25 output units and train it on a problem with up to 100 input patterns. Record your observations and explain why the network behaves in the way it does.

  7. Now try a problem where the number of output units is far more than the number of input patterns. What is noticeable about the final state of the network? Why has this happened?

  8. Now try presenting problems where the data are grouped in more complex ways (e.g. an ellipse with a cluster in the centre). Why does the Kohonen network not summarise the natural regularities in some of these input sets as well as others?

  9. The application of Kohonen networks is typically in mapping very high dimensional input spaces into 1 or 2 dimensions. Devise a problem with high dimensional inputs, write code to implement your own Kohonen network and conduct appropriate experiments.