TEACH TEACHNEURAL Julian Clinton Jan 1990 Modified A.Sloman 7 Mar 2000 Introduction To Poplog-Neural and on-line documentation. This is a modified version of the Poplog-neural package. See TEACH NEURAL NOTE: some of these facilities are available with different procedure names in the Popvision library produced by David Young and available from the FreePoplog directory: http://www.cs.bham.ac.uk/research/poplog/freepoplog.html See the MLP library and documentation contained therein. CONTENTS - (Use g to access required sections) -- How To Use This Documentation -- Introduction -- Software Description -- The LIB Neural Environment -- The Teach And Test Library -- The Generic Accessor Library -- Specific Network Types -- Rebuilding Poplog-Neural -- Using X Windows -- Linking C libraries -- Invoking Poplog-Neural -- Invoking Poplog-Neural With Graphical User Interface -- . X Windows -- How To Use LIB NETTRAINING -- Creating New Datatypes -- Using New Datatypes -- How To Use This Documentation -------------------------------------- The aim of this file is to teach you how to use the Poplog-Neural package rather than what to use neural networks for. It describes how the software is organised and how to make use of the various parts. The best way to achieve this is to read about the various parts of Poplog-Neural and then move on to the examples which aim to give a suitable way of working. NOTE added by A.Sloman, 29 Sep 2000. The original version of this assumed that you would run popneural code in the framework of a saved image with a special interface. The saved image is no longer set up by default, and in any case users are more likely to want to link the popneural libraries into their own programs. This teach file has been modified to suit such users. For most such users it will suffice to start with the section on How To Use LIB NETTRAINING -- Introduction ------------------------------------------------------- LIB Neural is a software package providing Poplog users with the facilities to: (a) teach themselves about neural networks (b) create different networks and train them (c) create new datatypes (e.g. sets, number ranges etc.) and example sets which can be applied to a neural network (d) display information about the state of a network. (e) allow networks to be embedded in programs and treated as standard procedures -- Software Description ----------------------------------------------- Poplog-Neural can be thought of as a collection of libraries collected into a single tool. The tool has the structure: 1. Poplog-Neural Environment Library (popneural) [Not relevant to this package] 2. Network Teach And Test Library (nettraining) -! 3. Generic Accessor Library (netgenerics) ! +- networkdefs 4. Specific Network Types (backprop, complearn) ! -! The aim has been to produce a system which can have new network architectures added to it without requiring a large amount of re-writing of the software (see HELP *NEWNETS for a guide to adding new network types). -- The LIB Neural Environment -------------------------------------- The environment is a menu-driven system with menus for creating networks, datatypes and examples, teaching and testing networks, altering environment options and getting help. The Poplog-Neural environment is documented in REF *NEURALINDEX and *POPNEURAL. It is no longer setup by default. In this version of the package it is assumed that the popneural library will be invoked from the editor and portions compiled as needed. I.e. the "environment" described here is not necessarily avaiable, though some of the files remain available as they contain useful documentation. -- The Teach And Test Library ----------------------------------------- This library provides a set of generic functions for teaching and testing networks. These functions are documented in the file REF *NETTRAINING. -- The Generic Accessor Library --------------------------------------- This library provides a set of network-independent accessor functions and a facility to support the addition of new network types. The functions are described in REF *NETGENERICS. -- Specific Network Types --------------------------------------------- Network types are stored as libraries so they can be loaded and used without the overheads of the other libraries being required. The libraries provided by the current release of Poplog-Neural are: (1) backprop - back-propagation networks (REF *BACKPROP) (2) complearn - competitive learning networks (REF *COMPLEARN) -- Rebuilding Poplog-Neural ------------------------------------------- [This section is no longer relevant. Skip to end of section.] When you receive Poplog-Neural, you should rebuild the saved images you are supplied with. You may also need to rebuild the system when you upgrade to a different version of the operating system. The base command for rebuilding Poplog-Neural is: neural -m (UNIX) MKNEURAL (VMS) Certain additional arguments can be provided (described below). -- Using X Windows ---------------------------------------------------- [This section is no longer relevant. Skip to end of section.] You can include an X user interface by using the "-x" option: neural -m -x (UNIX) MKNEURAL -X (VMS) The look and feel of the Neural you build will depend on the Poplog on which it is being built. By default, Poplog is built using the OPEN LOOK widget set on Sun workstations (Sun 3 and SPARCstations) and using Motif on all other platforms. See the Poplog Administrators Guide for more information on rebuilding Poplog with different X widget sets. Note that to build an X version of Neural may require you to setup some environment variable used by the operating system to locate the required libraries. For example, on Sun workstations the environment variable LD_LIBRARY_PATH may have to be defined or extended to point to the appropriate library directory e.g.: setenv LD_LIBRARY_PATH /usr/openwin/lib Normally, Poplog should have the required information defined within it. You can check which libraries Poplog will try to use by starting up Pop-11 and checking the value of XLINK_LIBS: % pop11 Sussex POPLOG etc. Setpop : XLINK_LIBS => ** [-L/usr/lib/X11R4 -L/usr/lib/Motif1.1 -lXm -lXt -lX11] -- Linking C libraries ------------------------------------------------ Instructions for compiling the C libraries SEE $poplocal/local/neural/src/c/* Select one of the compall* scripts that is suitable for your environment. E.g. on most unix systems compall.gcc or compall should work. You will have to set the environment variable $popneural Run the script with an argument that corresponds to the current system, for which there should be a matching sub-directory in $popneural/bin E.g. if you are on a PC running linux you could do setenv popneural (...whatever...) then compall.gcc pclinux This should put some compiled files in $popneural/bin/pclinux For a new machine and operating system you may have to create a new directory, and run compall.gcc with a different argument. Then edit $popneural/lib/cload.p to make sure that it copes with you machine and operating system. -- Invoking Poplog-Neural --------------------------------------------- [This section is no longer relevant] Poplog-Neural can be invoked from within any of the Poplog languages or direct from the UNIX or DCL prompt. To start the system from the operating system prompt, type: neural (UNIX) or NEURAL (VMS) This will start Poplog-Neural in standard terminal mode. -- Invoking Poplog-Neural With Graphical User Interface --------------- [These sections are no longer relevant] -- . X Windows -------------------------------------------------------- [This section is no longer relevant] Ensure that you are running an X11R4 server on your workstation console or X terminal and in a shell or XTerm window, type: neural -x This will invoke Poplog-Neural using X windows. The "look and feel" of the interface will depend on how the Poplog which Neural uses was linked. By default, Sun 3 and SPARCstation Poplogs are linked to use OPEN LOOK. On all other platforms, Motif is the default. -- How To Use LIB NETTRAINING ----------------------------------------- Load the neural net training library: lib nettraining; To do this you will need to have defined the environment variable $popneural and have added '$popneural/lib' to -popuseslist- and -popautolist-. NOTE: all this is done automatically by the command uses neural; Before going any further, we need to look at what our data consists of. In this example, which is teaching a network the exclusive-OR rule, the data is simply going to be a set of boolean values of the inputs and target outputs. (The exclusive-OR problem was shown to be impossible for a single layer network to learn in the late '60s and contributed to the temporary loss of interest in neural-like networks among many research groups). The exclusive-OR rule table looks like: Input 1 Input 2 Output -------------------------- false false false true false true false true true true true false and the example data can be specified as: vars xor_list = [[false false false] [true false true] [false true true] [true true false]]; We also need to specify what each example in the set looks like. This template is also a list of lists, usually one for each field in the example. Here there are two input fields followed by one output field (it is also allowable to ignore a field by entering "none" or having the data presented at both input and output units by typing "both"). Each entry is of type "boolean". This can be specified as: vars xor_template = [[in boolean input1] [in boolean input2] [out boolean output]]; The third item in each list is an optional field name. It is a good habit to use this since it makes a print out of an example set easier to understand. Create the example set with: nn_make_egs("xor_set", xor_template, xor_list); and then generate the raw data: nn_generate_egs("xor_set"); -nn_generate_egs- creates the data and converts the high level datatypes (e.g. boolean) into numbers suitable to be applied to the network (in this case the numbers 0.0 for false and 1.0 for true). To prove that this has been done, we can display the contents in one of the example set record slots: vars eg_rec = nn_example_sets("xor_set"); eg_targ_data(eg_rec) => ** This shows that the output data to be presented to the network consists of four groups of data, each group being 1 element long. The input data consists of a similar array but each group contains two items of data. eg_in_data(eg_rec) => ** It can also be useful to show the contents of an array using *ARRAYVECTOR. For example: arrayvector(eg_targ_data(eg_rec)) => ** arrayvector(eg_in_data(eg_rec)) => ** Having created the example set, we can create a network to learn the rule. From looking at the examples, we can see that the network will need 2 input units and and 1 output unit. We will also give the network a hidden layer of 2 units (a hidden layer is a layer of units which are neither input or output units) since this is the topology which has been shown to be capable of learning the exclusive-OR rule (shown below). Output O / \ / \ / \ O O | \ / | | \ / | | X | | / \ | | / \ | O O Input 1 Input 2 The type of network used is very important. In this case, our examples have both input and output values which implies using a back-propagation network. A back-propagation network learns by comparing the actual response for a given input with the target response and altering the strengths of the connections between the units to reduce the response error. (This compares with a competitive learning network which attempts to 'organise' the input data on its own by encouraging units to respond strongly to patterns with some common characteristics and not to others.) The is a library which implements back-propagation networks called BACKPROP (see REF *BACKPROP). This will have been loaded along with the NETTRAINING library. Create the network with: make_bpnet(2, {2 1}, 2.0, 0.2, 0.9) -> nn_neural_nets("xor_net"); This creates a network with 2 input units, 2 units in the hidden layer and 1 in the output layer and stores it in. The weights (strength of connection between the units) are each set to a random number between -1.0 and +1.0 (a range of 2.0). The last two figures are the learning rate, eta, and the momentum, alpha (see "Parallel Distributed Processing", McLelland & Rummelhart, for a detailed description of these parameters). We can print the weights of a back-propagation network with PR_BPWEIGHTS: pr_bpweights(nn_neural_nets("xor_net")); WEIGHTS bias 1 2 Level 2 unit 1: 0.57 -0.74 -0.56 Level 1 unit 1: 0.05 0.90 0.60 unit 2: 0.48 0.88 0.73 This shows that the strength of the connection between the first unit in hidden layer 1 and the second input unit is 0.60. vars results = nn_test_egs("xor_set", "xor_net", false); results => ** [[false] [false] [false] [false]] (You may get different results from this.) What we actually want is ** [[false] [true] [true] [false]] We want the network to learn to respond correctly to the input which is done with: nn_learn_egs("xor_set", "xor_net", 1000, true); We can check how well the network has learnt by applying the example set again: nn_test_egs("xor_set", "xor_net", false) => ** [[false] [true] [true] [true]] (Again you may find that this does not correspond with your results.) Although in this case the response was not correct, we can see that the network is beginning to learn the correct responses. An important point to note is that the network you have created may not learn the correct responses all the time. This is due to the network becoming stuck in a local minimum where weight changes made to make one response correct are cancelled out by other weight changes. In the majority of cases however, if we continue the learning process, the network will end up being correctly trained. nn_learn_egs("xor_set", "xor_net", 5000, true); nn_test_egs("xor_set", "xor_net", false) => ** [[false] [true] [true] [false]] ;;; hooray ! We can see how the weights have changed from before: pr_bpweights(nn_neural_nets("xor_net")); WEIGHTS before training WEIGHTS (after training) bias 1 2 bias 1 2 Level 2 Level 2 unit 1: 0.57 -0.74 -0.56 unit 1: -4.11 -9.66 8.96 Level 1 Level 1 unit 1: 0.05 0.90 0.60 unit 1: -6.69 4.36 4.36 unit 2: 0.48 0.88 0.73 unit 2: -2.76 6.28 6.28 -- Creating New Datatypes --------------------------------------------- There will be many occasions when the datatypes defined by NETGENERICS are not sufficient. For example, suppose we want to train a network to accept percentages of red, green and blue in a given light source and return the colour that is represented by that light source. The NETGENERICS library provides functions to allow the user to define new datatypes. The "percent" datatype is going to be an integer somewhere between 0 and 100. This can be created using the function NN_DECLARE_RANGE: nn_declare_range("percent", 0, 100); (If we had wanted to have the result returned as a real instead of an integer, the third argument would have been replaced with 100.0. The function which takes results from the output of the network and converts it to a number within the range also coerces the result to be of the type of the number specified as the upper bound (see REF *NUMBERS/number_coerce).) The set of colours can be defined using -nn_declare_set- (see REF *NETGENERICS). This takes a set name and a list of set members. For example: nn_declare_set("colour", [red orange yellow green blue indigo violet]); -- Using New Datatypes ------------------------------------------------ Having defined the required datatypes, we can create a couple of example sets, one for teaching and one for testing. Both will take three inputs (the red, green and blue values) and a single output (the colour). nn_make_egs("colour_set", [[in percent red] [in percent green] [in percent blue] [out colour colour]], [[100 0 0 red] [40 40 20 yellow] [0 100 0 green] [0 0 100 blue] [40 0 60 violet] [30 0 70 indigo]]); nn_generate_egs("colour_set"); Finally we can create the network which will attempt to learn these examples. Before we do this, we will need to find out how many input and output units are needed to represent the different datatypes being used. This can be done in two ways: 1. By using the function -nn_units_needed-. This takes a list of datatypes and returns the number of units required to map the datatypes onto a network. The input has three sets of "percent" datatype and the output is a single "colour", so: nn_units_needed([percent percent percent]) => ;;; for input ** 3 nn_units_needed([colour]) => ;;; for output ** 7 2. By accessing the example set record itself. There are two slots in the record accessed by the functions -eg_in_units- and -eg_out_units-. Once we have retrieved the example sets from the -nn_example_sets- table, we can find out how many input and output units are required by the example data: vars eg_rec = nn_example_sets("colour_set"); eg_in_units(eg_rec) => ** 3 eg_out_units(eg_rec) => ** 7 We will make our network with 3 input and 7 output units. There will be one hidden layer with 4 units: make_bpnet(3, {4 7}, 2.0, 0.2, 0.9) -> nn_neural_nets("col_net"); pr_bpweights(nn_neural_nets("col_net")); WEIGHTS bias 1 2 3 4 5 6 7 Level 2 unit 1: 0.25 0.30 0.84 0.95 -0.42 unit 2: -0.13 -0.56 0.36 0.19 0.45 unit 3: 0.44 1.00 -0.98 0.55 -0.53 unit 4: -0.01 0.73 0.04 0.78 0.67 unit 5: -0.67 -0.22 -0.60 -0.09 0.60 unit 6: -1.00 0.22 -0.99 -0.60 -0.10 unit 7: -0.93 -0.95 -0.43 0.02 -0.58 Level 1 unit 1: -0.48 0.30 -0.46 -0.41 unit 2: 0.57 -0.41 -0.98 -0.41 unit 3: 0.45 0.04 0.44 -0.64 unit 4: 0.48 -0.73 0.80 0.04 Now the example set "colour_set" can be learnt by the network: nn_learn_egs("colour_set", "col_net", 5000, false); nn_test_egs("colour_set", "col_net", false)=> ** [[red] [yellow] [green] [blue] [violet] [indigo]] Again, you may find that these results don't agree completely with yours. For example, the last result, "indigo", may actually be "violet" because the difference in values between "indigo" and "violet" are not as great as they are with the other colours. This means the network will have to be trained further in order to be able to discriminate between them. --- Copyright Integral Solutions Ltd. 1990. All rights reserved. --- --- Copyright Integral Solutions Ltd. and the University of Sussex, 2000 All rights reserved. --- --- $poplocal/local/neural/teach/teachneural --- Copyright University of Birmingham 2000. All rights reserved. ------