REF BACKPROP David Young Sep 1990 Updated J. Clinton Aug 1992 Copyright University of Sussex and Integral Solutions Ltd. All Rights Reserved >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<<<< FUNCTIONS & VARIABLES >>>>>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<<<< IN LIB BACKPROP >>>>>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>>>>>>>>>> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< CONTENTS - (Use g to access required sections) -- Before Loading The Library -- Creating A Network -- Network Parameters -- Network Response -- Training Networks -- Checking Array Consistency -- Printing Information -- Saving And Loading Networks -- Before Loading The Library ----------------------------------------- All the libraries available in Poplog-Neural make use of a number of environment variables. These are: 1. HOST_TYPE Set to be the name of the machine being used e.g. sun3, sun4, bobcat, vax etc. 2. NEURAL_F77 Specifies whether a Fortran 77 compiler is available. This should be set accordingly to "yes" or "no". These should be defined in your .login (UNIX) or LOGIN.COM (VMS) file. -- Creating A Network ------------------------------------------------- make_bpnet(NIN, NUNITS, WTRANGE, ET, AL) -> NETWORK [procedure] This sets up a back propagation network for subsequent use. Here NIN is the no of input units, NUNITS is a vector giving the no of units in each layer above that, starting with the lowest hidden layer (if there is one) and finishing with the output layer, WTRANGE is the range over which the random initial weights are to be distributed, ET is the learning rate (roughly eta in R & McC) and AL is the momentum term (alpha in R & McC). For example make_bpnet(30, {15 5}, 2.0, 0.05, 0.9) -> network; will return a network with 30 input units, one layer of 15 hidden units, 5 output units, initial weights distributed between -1.0 and 1.0, a learning rate of 0.05 and a momentum of 0.9. You can change eta and alpha later if you want, e.g. 0.1s0 -> network.bpeta; 0.8s0 -> network.bpalpha; -- Network Parameters ------------------------------------------------- bpeta [procedure] This accessor can be used to change the learning rate of a network. The value assigned should be a single precision real. For example 0.1s0 -> network.bpeta; bpalpha [procedure] This accessor is used to change the learning momentum of a network. Like -bpeta-, the value must be a single precision real e.g. 0.8s0 -> network.bpalpha; bpweights [procedure] bpwtchange [procedure] bpbiases [procedure] bpbschange [procedure] bpactivs [procedure] These all take a network as argument and return lists of arrays (one array for each level of the network) containing, respectively: the current weights the last change to the weights the current biases the last change to the biases the current activations (if the last updating call was to -bp_response-) or the current error signals (if the last updating call was to -bp_learn- or -bp_learn_set-). You cannot use the updaters of these procedures to change the network, nor can you update the lists they return. The results will be meaningless if you try. To change the weights and biases in the network, you can update the elements of the arrays individually. -- Network Response --------------------------------------------------- bp_response(INVEC, NETWORK, OUTVEC) [procedure] This forward propagates an input through the network, without doing any learning. Here INVEC is the input vector, MACHINE is something returned by -make_bpnet-, and OUTVEC receives a copy of the output vector. INVEC and OUTVEC must be arrays of real numbers, created using ARRAY_OF_REAL, not NEWARRAY. (ARRAY_OF_REAL takes the same arguments as NEWARRAY - a boundslist and an optional initialiser - but makes a Fortran compatible array.) INVEC must be at least as big as the no of input units stated for the network, and OUTVEC must be at least as big as the no of output units. bp_response_set(INPUT_SET, NETWORK, OUTVECS) [procedure] bp_response_set(STEPSIZE, INPUT_SET, NETWORK, OUTVECS) [procedure] This forward propagates a set of inputs through the machine, without doing any learning, and stores the results. INPUT_SET is the set of input vectors or stimuli. If there are 3 arguments, INPUT_SET must be a two-dimensional array of separate stimuli, with one column (in matrix convention) for each stimulus. In this case, if there are NIN input units and you want to have NSTIM different stimuli, then the boundslist should be [1 ^NIN 1 ^NSTIM] and INPUT_SET(I,J) is the I'th element of the J'th stimulus. In the second form, with 4 arguments, INPUT_SET must be a one-dimensional array of stimuli, which may be stored successively, with spaces in between, or overlapping one another. STEPSIZE gives the separation between the start points of the individual stimuli. Each stimulus must be a contiguous set of NIN values in the array, starting at element 1 + (I-1) * STEPSIZE for the I'th stimulus in the array. If STEPSIZE is equal to NIN, then the stimuli do not overlap and there is no real difference from the first way to call the procedure. If STEPSIZE is 1, then the input is treated as a time series onto which the network has a moving window which samples every possible sub-sequence. OUTVECS is an array such that OUTVECS(I,J) will receive the I'th element of the J'th output vector, and its boundslist will be [1 ^NOUT 1 ^NSTIM] if there are NOUT output units in the network and NSTIM stimuli to process. If the procedure is called with a 2-dimensional INPUT_SET, then the second dimensions of INPUT_SET and OUTPUT_SET must match. If the procedure is called with a 1-dimensional INPUT_SET, then the length of INPUT_SET must be at least (NSTIMS-1) * STEPSIZE + NIN, where NSTIMS is obtained from the boundslist of OUTVECS. -- Training Networks -------------------------------------------------- bp_learn(INVEC, TARGVEC, NETWORK) [procedure] This does the learning bit: it updates the weights in MACHINE. It must follow a call to -bp_response-. TARGVEC must be an array of reals (created using -array_of_real-), at least as long as the no of output units, giving the target vector corresponding to the input vector used in the last call to -bp_response-, and INVEC must be that input vector. bp_learn_set(STIMS, TARGS, NITER, CYCLE, NETWORK, OUTVEC) [procedure] bp_learn_set(STEPSIZE, STIMS, TARGS, [procedure] NITER, CYCLE, NETWORK, OUTVEC) This allows a number of cycles of forward and backward propagation to be done in one call (it is more efficient than repeated alternate calls to -bp_response- and -bp_learn-, which is effectively what it does). In the first form, STIMS is a 2-dimensional array of stimuli; if there are NIN input units and you want to have NSTIM different stimuli, then its boundslist should be [1 ^NIN 1 ^NSTIM] and STIMS(I,J) is the I'th element of the J'th stimulus. Similarly TARGS is an array such that TARGS(I,J) is the I'th element of the J'th target, and its boundslist will be [1 ^NOUT 1 ^NSTIM] if there are NOUT output units. NITER is the no of iterations to carry out; it can be greater or less than the no of stimuli in the array. If CYCLE is true, then the stimuli are presented cyclically, going through the STIMS and TARGS arrays in sequence; if it is false the stimuli are picked out at random. MACHINE is a network set up as before, and OUTVEC receives a copy of its output vector, and so must be an array with at least NOUT elements. (OUTVEC is not a lot of use - it's effectively workspace for the program and for compatibility.) It's not necessary to call -bp_checkvecs- before calling -bp_learn_set-. In the second form, STIMS is a 1-dimensional array of stimuli, useful for when the inputs are to be treated as a time series. See -bp_response_set- for a description of how this works in conjunction with STEPSIZE. bp_backinvec(NETWORK, INVEC) [procedure] This is a rather arcane procedure that can be useful in unusual architectures. If two machines are 'stacked', one above the other (for instance if you need to mess about with the intermediate representation, or you want several low-level machines to feed one higher level one), then it is easy to forward propagate by using the output vector from the lower network as the input vector for the higher one. -bp_backinvec- allows you to back-propagate across the gap: it back-propagates the error signal into the input vector in such a way that the latter may be used as the target vector in a call to -bp_learn- with a lower network. The procedure must be used after a call to -bp_learn- on MACHINE, and INVEC must be the input vector used in that call to -bp_learn- and the preceding call to -bp_response-. It is updated to provide the new target vector for a lower network. -- Checking Array Consistency ----------------------------------------- bp_checkvecs(NETWORK, INVEC, OUTVEC, TARG) [procedure] This is a routine that simply mishaps if INVEC, OUTVEC or TARG isn't big enough to be compatible with MACHINE. It is sensible to call it before calling -bp_response- and -bp_learn-. -- Printing Information ----------------------------------------------- pr_bpweights(NETWORK) [procedure] This simply prints out all the weights of a network. pr_bpactivs(NETWORK) [procedure] After a call to -bp_response- or -bp_learn_set-, this prints out all the activations of a network. After a call to -bp_learn-, it prints out all the error signals. -- Saving And Loading Networks ---------------------------------------- bp_save(FILENAME, NETWORK) -> RESULT [procedure] Saves NETWORK (a back propagation network record) to disk. The filename should have a '.net' suffix. Returns if the network was saved ok or false otherwise. bp_load(FILENAME) -> NETWORK [procedure] This procedure takes a filename (a string) or a device and reads the file from that file/device. A back propagation network record is returned if the file was successfully read otherwise is returned. --- $popneural/ref/backprop --- Copyright University of Sussex 1990. All rights reserved. ----------