BRITISH ASSOCIATION ANNUAL FESTIVAL - PRESS PAPERS FORM SPEAKER'S NAME: Surname: Sloman Forenames: Aaron Title: Professor of Cognitive Science and Artificial Intelligence NAME OF ORGANISATION: The University of Birmingham Address: School of Computer Science The University of Birmingham Edgbaston Birmingham B15 2TT TITLE OF PAPER: What kinds of machine can have emotions? DATE OF PAPER: Thursday 12th Sept 1996 TIME OF PAPER: 1645 (Room: Poynting Lecture Room, Physics) NOTE 1 The text that follows is a summary not a transcript. The presentation at the conference made considerable use of diagrams. See http://www.cs.bham.ac.uk/~axs/misc/emotions.mit96.slides.ps NOTE 2: In this plain text version {\em .... } indicates that the text enclosed should be italicised (emphasised). Also G\"odel represents the name "Godel" with an umlaut over the "o". The precis follows. ======================================================================= WHAT KINDS OF MACHINE CAN HAVE EMOTIONS? Aaron Sloman School of Computer Science The University of Birmingham http://www.cs.bham.ac.uk/~axs ABSTRACT: Work in Artificial Intelligence (AI) provides a new framework for thinking about how minds work. The talk will show how the framework can accommodate motivational and emotional aspects of mind. This is done by relating AI to the study of design space, niche space and their relationships. We describe three levels of sophistication in designs for behaving systems {\em reactive}, {\em deliberative} and {\em meta-management} and we outline ways in which they may be able to explain both some capabilities which we share with other animals and some distinctively human capabilities, including forms of conscious and ``perturbant'' emotional states. Besides its theoretical interest this is potentially of considerable practical importance in helping us understand ourselves and perhaps helping us to remedy various things that can go wrong. So far there is no evidence that these architectures and mechanisms depend on extensions to contemporary physics such as the quantum gravity theory proposed by Penrose. INTRODUCTION This talk is partly intended to be a rebuttal of the sorts of arguments being put forward by Professor Roger Penrose, who will also be talking at the BAAS conference in the Brains Minds and Consciousness session on the same day. He and others attack the so-called "Strong AI" thesis, one version of which claims that the sorts of systems being developed within the discipline of AI can in principle have mental states and processes, such as believing, desiring, deciding, understanding or being conscious. (There are other versions which are less interesting.) [Note: "AI" now often used as a name in its own right was originally an abbreviation for "Artificial Intelligence". This name is misleading, however, since the discipline has always been as much concerned with modelling {\em natural} intelligence in humans and other animals as with creating useful "smart" artefacts, though some researchers have concentrated more on modelling humans (e.g. H.A.Simon and A.Newell) others more on designing artefacts (e.g. John McCarthy, who invented the name).] Some of those who attack AI do so on the basis of a famous mathematical theorem by Kurt G\"odel, his second incompleteness theorem. The theorem is mathematically complex and its interpretation in this context disputable. The anti-AI argument depends essentially on the claim that human mathematicians can see the truth of a complex mathematical formula that could not be derived by any formal computation satisfying certain standards of computational rigour. Many reputable mathematicians and logicians reject the arguments by Penrose and others based on this theorem. My own refutation of Penrose's version was published in 1992.[Sloman92] However, since very few human beings, and, as far as I know, no other animals, can understand G\"odel's theorem, it is very odd to use this esoteric mathematical capability as the basis for an attack on AI. At most it would prove that we cannot use AI techniques to produce artificial mathematicians who are able to reason about infinite sets of numbers. This would leave open the question whether we can produce robots that can perceive things, have beliefs, have desires, have intentions, make plans, learn things, solve problems, be conscious sometimes and unconscious at others, and be emotional at times, without being capable of doing advanced mathematics, like a young child and most adults. I propose that we should first look at these more basic and widespread mental capabilities, shared by all human beings, and perhaps many other animals, and see whether we can understand what sorts of architectures and mechanisms make them possible, before we try to explain or model sophisticated mathematical capabilities that we do not share with other animals and which are not obviously present in all human beings. It has recently become fashionable to focus on {\em consciousness}, as if it were a well defined clearly identifiable phenomenon about which it makes sense to ask questions like: How did it evolve? What are its functions? Which animals have it? What are its neural correlates? What sorts of physical machines can produce it? This is another topic on which Penrose has written much, claiming that "it" can occur only in physical systems whose operation cannot be explained by contemporary physics, but requires an extension which links quantum mechanics with gravitational forces. Unfortunately, although Penrose may be right that we need to extend modern physics in order to gain a full understanding of how the brain works (and there's nothing surprising in the suggestion that our current knowledge of the fundamental nature of physical reality is still incomplete), I do not think there is any well defined concept of consciousness around which it makes sense to focus the questions he and others wish to raise. Rather, typical adult human consciousness involves a collection of different capabilities, different subsets of which can occur in different people, in different animals, in newborn infants, in brain damaged people, in cases of senile dementia. Thus there is no single "it" about which we can sensibly asked how it evolved, what its function is, etc. (The tendency to think there is a single "it" arises out of deep philosophical confusions which I shall not discuss here.) THE DESIGN-BASED APPROACH TO THE STUDY OF MIND All this suggests that a sensible approach to the study of mind, including attempts to replicate or model mental phenomena in machines, may be to try to isolate different sorts of capabilities, show how they might have evolved, describe their functional roles in different animals (and robots), explain how they are implemented in animal brains, and investigate whether they could be implemented in machines of various kinds. This is obviously a multi-disciplinary research programme, requiring collaboration between biologists, brain scientists, psychologists, AI researchers and philosophers (like myself!). In attempting to model or replicate the phenomena, we should not necessarily restrict ourselves to any particular form of engineering technique: that would be silly. So, if necessary, we should combine digital computers, neural nets, various kinds of software engineering techniques, chemical mechanisms, and whatever else proves useful or necessary, including quantum gravity computers if such things are ever created. If powerful new information processing engines become available AI research of the future may look very different from AI research of the last fifty years. But that does not mean that we have to wait for such new developments doing nothing in the meantime. On the contrary, we can try to understand what the problems are, and see how far we can go towards solving them using mechanisms that we already understand. If we discover limitations in those mechanisms as a result of such efforts we shall be in a much better position to understand requirements for new mechanisms. My own view is that so far there is no clear evidence that new advances in physics are required for understanding or replicating typical mental phenomena. A fruitful framework for conducting this research is to explore "design space" and "niche space" and their relationships. "Design space" is the space of possible designs for systems, with different combinations of capabilities. "Niche space" is the space of possible combinations of requirements for behaving systems. It includes both what biologists study when they talk about the niche of an organism and partly what engineers study when they talk about the requirements against which a design is to be evaluated. Niche space and design space are both very complex structures, and the mappings between them are very complex. What are the dynamics of these spaces: what sorts of trajectories are possible within them? Which sorts can occur in individual development? (As far as I know, it is not physically or chemically possible for a human egg to evolve into a giraffe, no matter how its environment is manipulated.) Which sorts of trajectories require evolution across generations involving many individuals? Which can occur in a laboratory? Evolution also produces designs -- even though there is no designer or engineer, just natural selection. The notion of a design is an abstraction that has nothing to do with how the design was produced, or whether any agent intends it to serve any purposes. In that sense the design of a bee fits the requirement to pollinate plants. Similarly evolution produces niches: the design of one organism may form part of the niche for another. Thus the dynamics of niches and designs will be intimately related. One aspect of design that is familiar to engineers is the notion of an {\em architecture} within which various sorts of mechanisms (and sub-architectures) can be combined into an integrated functioning whole. This is not necessarily a physical architecture like the architecture of a building. For example a complex factory control system, or financial management system, or international information network, may have a software architecture, defined in terms of collections of interacting modules with different sorts of capabilities, and may include several different layers of abstraction, such as the different levels of transmission protocols on networks. Another important point which is not familiar to all engineers is that the processes within an architecture need not all be describable in terms of a fixed collection of numerical measures, such as voltage, current, torque, velocity, pressure, etc. In an information processing architecture many of the important changes are changes in structures that record information, such as sentences, logical formulae, decision trees, networks of linked structures, and so on. The changes need not all be changes in values of numerical variables, but can include all sorts of other changes, including the sorts of changes that we are now familiar with in databases, game playing programs, compilers, operating systems, word processors and the increasingly fashionable "software agents". Further, in such systems the architecture need not be static: software systems can dynamically reconfigure themselves to meet different demands, and they can learn how to do this on the basis of records of what works well in different contexts. Thus we can conceive of a space of possible designs for information based control systems. My conjecture is that the mind is such a control system [sloman93], or rather there is a class of possible minds, including minds of young children, of normal adults, or brain damaged individuals, and also many other animals, including chimps, monkeys, cats, dogs, nest building birds, and perhaps insects and single celled organisms. This notion of "mind" does not have any sharp boundaries, and that does not matter, for it's the divisions and discontinuities {\em within} the space that are interesting, and which we need to understand and explain, including explaining how processes of evolution or individual development can produce transitions across those design discontinuities. Understanding the characteristically human control system requires investigation of a complex architecture composed of layers with different evolutionary histories. I shall describe three different layers of architecture and then relate them to certain human emotional phenomena. A. REACTIVE ARCHITECTURES One kind of control architecture is purely reactive. Information is acquired through external sensors and internal monitors and propagates through and around the system, and out to effectors of various kinds. Everything happens automatically and in parallel because there are dedicated, coexisting circuits, some continuous and some digital, implementing condition-action rules, with various kinds of feedback. Such a system can react quickly because all processing is done in parallel. Competitions between sub-systems can be resolved either by weighted additive mechanisms where the appropriate behaviour is a combination of sub-behaviours, or by winner-take-all circuits where sub-systems tend to produce incompatible behaviours. This sort of architecture can work very well provided that the environment for which it has been developed (by evolution, or by an explicit designer) does not change much. If there is a lot of variability in the environment it may be that the survival rate for individuals is not very high, but that may not matter for the gene pool if very large numbers of individuals can be produced cheaply and enough survive in each generation. I suspect that some animals are {\em entirely} like that (e.g. insects), some are {\em largely} like that (e.g. other animals whose behaviour is mostly genetically determined), and chimps and humans are {\em partly} like that, with additional architectural layers providing additional flexibility. B. DELIBERATIVE ARCHITECTURES. In a reactive architecture, everything happens automatically: whenever the conditions for an action are satisfied the action is triggered, though its effects may be suppressed by other parts of the system. By contrast, a deliberative architecture allows the explicit construction of alternative possible future actions, including novel actions, along with processes of evaluation and comparison, leading to the selection of one option, as a plan (possibly a partial plan). This requires use of stored knowledge about the environment, about the agent's capabilities, about preconditions and effects of actions. In general the process of finding a suitable plan, or solution to a problem requires a search in a space of possible structures. This can include symbolic trial and error searching, which may be cheaper, quicker, and safer than trying out options in the real world. This sort of architecture requires a re-usable workspace within which the temporary structures representing possible actions can be built and evaluated, until a selection is made. The same workspace can then be used for another problem. The use of such a workspace is therefore inherently serial, even if it is implemented using parallel mechanisms, such as neural nets. The global store of knowledge required for constructing novel solutions may also be implemented using a highly parallel mechanism, but that does not mean that it can answer lots of questions in parallel: it too may be restricted to a serial mode of functioning. For these and other reasons that are too complicated to go into here, the overall functioning of a deliberative architecture in which novel temporary structures are built and selected will be inherently resource limited. Only a limited number of things can occur at a time and the system cannot be made faster by multiplying workspaces, whereas in a reactive system additional parallel circuits dedicated to performing specific tasks (e.g. visual sensing, auditory sensing, touch sensing, posture control, control of digestion, etc.) can be added to the architecture if needed. In a hybrid system combining both architectures, which I conjecture is to be found in many animals, it may be possible for useful new structures to be moved or copied from the temporary workspace in the deliberative architecture into dedicated reactive circuits. This seems to be a common feature of training of athletes, training of musical performers, learning to drive a car, and many forms of intellectual training which require rapid recognition of structures and retrieval of solutions to problems, as in mathematical expertise and learning to read. I suspect that insects do not include a deliberative mechanism (though I am not an expert on insects). I am sure that some other animals do (e.g. chimpanzees and bonobos) though I don't know how far they compare with human deliberative capabilities. Detailed investigation may show wide variations in the deliberative capabilities of different sorts of animals, related to their different capabilities to perceive the environment, store abstract representations for future use, manipulate representations to solve problems, and so on. A great deal of work in AI has been concerned with the study of deliberative mechanisms, though there has also been a lot of interest in neural nets and condition-action systems which can be used for reactive systems (sometimes referred to as "behaviour-based" systems). However, there has not been quite so much AI work on multi-purpose deliberative architectures embedded in a complex and changing environment with real-time constraints, although many software engineering projects need to meet those requirements. In such contexts, the use of the workspaces must be interruptable (e.g. because of the need to rapidly drop whatever you are doing and thinking about in order to decide how to escape from a fast approaching serious danger, when the answer isn't sufficiently obvious to be settled by the reactive system). To cut a long story short, these (empirical and engineering design) arguments about deliberative mechanisms lead to a theory about the need for both (a) mechanisms that can interrupt and reorganise what's going on in the reusable workspace and also (b) filtering or "gating" mechanisms, so that what is able to interrupt and reorganise depends on some comparison of importance and urgency both of the existing tasks and the potential interrupter. We have here the seeds of an idea that sometimes these mechanisms can "go wrong", producing too many interrupts and diversions so that important and urgent tasks are not successfully completed. Many other kinds of malfunctions are possible, including distorted evaluations of the relative merits of different options, failure to use available knowledge in constructing new solutions, unintended modifications of complex plans and other structures so that they no longer work properly, and many more. This is one of many ways in which the AI approach to designing intelligent systems and work on psychotherapy and counselling could interact fruitfully. (Unfortunately pressures on teachers and researchers make it increasingly difficult to find time for new types of interdisciplinary collaboration.) C. THE META-MANAGEMENT LAYER Because the re-usable workspace is a powerful resource that has to be shared between different competing goals and needs which in some environments can change rapidly, issues arise about how to schedule its use when there are competing demands, for instance when conflicts arise between desires to eat, to avoid an impending danger, to get warmer, to find a mate, to protect a helpless infant, to investigate a puzzling phenomenon, and so on. With colleagues at Birmingham, and especially a former PhD student, Luc Beaudoin, I have been exploring the notion of a third level of architecture, which, in evolutionary terms, would be even more recent, and possibly a lot more rare than the reactive and deliberative layers. [Beaudoin94] This involves a "meta-management" system which can (to some extent) monitor the strategies and behaviour of the deliberative and reactive systems (and possibly also the meta-management system itself) and take corrective action when the individual decisions do not seem to be producing an overall state that is valued highly. Meta-management mechanisms could simply be a specialised subset of the deliberative mechanism with the ability to monitor and categorise and act upon the states of the deliberative mechanism. Thus it might discover and remedy weaknesses in the deliberative strategies. An example might be learning that in some cases problems can be detected early in the process of plan formation rather than later in the processing of running a plan. Many programmers develop their programming capabilities in this way, but it seems to be a far more general phenomenon. Another example many people will find familiar is noticing that one is switching attention between tasks too frequently, with consequent loss of efficiency. Corrective action might involve deciding to ignore interrupts and new motives for a time. Such control at the meta-management level is not perfect: we sometimes decide, and want, to think about X, but are continually drawn back to thinking about Y, perhaps because filters in the interrupt mechanisms mentioned above are not using optimal strategies, or because something has gone wrong through brain damage, chemical imbalances, or whatever. Some characteristically human emotional states seem to involve such partial lack of control of attention. I call these perturbances, below. The third level explains the fact that you can not only look at things around you and do things, but you can also attend to what you are looking at, think about what you are paying attention to, and notice aspects of your experience, such as how what you see varies as you move, without which much art would be impossible. I conjecture that this third architectural layer (which, like the other two, we still only partially understand) is to be found in humans but probably not at birth: i.e. it seems to develop within an individual, though the mechanisms that make it possible, and its general form, may be genetically determined. E.g. a very young child may be unaware that its parent has successfully diverted its attention away from some desirable object, and therefore the child will not redirect attention to that object. A few months or years later the trick no longer works and parents have to be far more subtle in their control strategies, because the child can remember that it had an important unfulfilled goal a short time ago. The capabilities of meta-management can be enormously enhanced through the development of a culture that includes a language for talking about mental states, thereby ensuring that it is not necessary for each individual to learn entirely unaided how to categorise and control his/her deliberative processes. In some other mammals (bonobos? monkeys?) the meta-management architecture may also be present, though perhaps with reduced functionality, partly because of the lack of a human-like language. TOWARDS A FUNCTIONAL THEORY OF `QUALIA' Many scientists and philosophers have been puzzled by what is variously referred to as the contents of subjective experience, qualia, what it is like to be something. This is supposed to be specially "hard" for science to explain, and some have argued that only new kinds of physics (e.g. quantum gravity theory) can explain such things, while others have claimed that they can already identify physiological mechanisms in the brain that are a sufficient explanation. There are also debates about whether these phenomena can occur in computer based robots or some other kind of intelligent future machine. My own view is that there is a mixture of philosophical confusion and scientific naivety in much of this discussion but that insofar as there is a well defined problem the answer can be provided within the framework of the design standpoint illustrated by this paper. In particular, we can consider designs in which meta-management processes are able, in some circumstances, to inspect some of the internal states of other mechanisms, including some of the intermediate states of sensory processing sub-systems (which can themselves have very complex architectures including several layers of interpretation mechanisms using more and more abstract virtual machine structures), and some of the states of the deliberative and control systems. The inspection may always be only partial, and may always involve an element of interpretation or translation (e.g. from a low level sensory notation to a formalism that can be used by the meta-management system). But there is nothing mysterious about it. Self-monitoring, though much simpler in form, is commonplace in computer operating systems and it may turn out that extensions of those techniques will suffice for self-monitoring in more intelligent machines. This internal monitoring of intermediate sensory stores happens for example, when you learn to see that besides the rectangular shape of the table in front of you there is also a skewed shape (a parallelogram) which is not out there but is part of an intermediate representation within you, determined by the structure of the viewpoint relative optic array: i.e. it changes shape as you move. Without this internal monitoring capability, "realistic" 2-D painting or drawing of 3-D objects would be impossible. The biological functions of the "qualia" are real, but too complex to describe in detail here. E.g. they can play a role both in individual learning, and also in communications where one agent tries to help another "debug" an internal process: "Look just a teeny bit to the left of the large green triangle, dear.... then you'll see it" where there is no green triangle out there, but from the current viewpoint there's a triangular region caused by e.g. the way two sloping walls and a floor bound a visible part of a rectangular lawn). On this model there are many different kinds of qualia (contents of self-monitoring states), and they are to be explained at a functional level in terms of the architecture that makes them possible, and at a physical or physiological level in terms of the mechanisms used to implement the architecture, which may be different in different organisms or machines. The three layers in the proposed architecture, the automatic reactive processing layer, the management layer which can construct and contemplate options in advance, and the metamanagement layer which can monitor, evaluate, and redirect high level internal actions, may each add new kinds of capabilities compared with earlier, simpler systems. Within the space of possible designs we can expect to see many discontinuities, depending on which mechanisms and which functions are present. Trying to treat all cases as part of a single (smooth) continuum is therefore seriously misleading, as it diverts attention from the important task of identifying the discontinuities and the mechanisms underlying them. WHAT DOES ALL THIS HAVE TO DO WITH EMOTIONS? There are some aspects of emotion that we share with many other animals and depend on processes in the reactive subsystems of the brain. These include being startled, terrified, thrilled, distressed by extreme pain, disgusted by horrible tasting food, sexually aroused, and no doubt many more. I am not an expert on brain processes but I understand that the limbic system plays a major role in these processes. I suspect that when we have a more fine grained understanding of the functionality of the reactive architecture and the way it interacts with deliberative and meta-management mechanisms we shall be able to produce a more systematic taxonomy of control states including emotions, moods, desires and various kinds of pleasures and pains. However, there are some sorts of emotional states that (it seems) cannot occur in rats or cats, but are commonplace in human beings, and may require something other than a purely reactive architecture. These are the emotions that poets, novelists, playwrights and gossips are interested in. Examples are being humiliated in front of a rival, ashamed of what one has done to a friend, awe-struck by a magnificent building, thrilled at winning a gold medal, or obsessed with a desire for revenge or promotion? In [Wright96] the example of grief is analysed in some depth and related to the sort of three-layered architecture described here. Months after a road accident that killed his child a grieving father may find it difficult to avoid thoughts about the child, what might have been done to avoid the accident, what the child would have been doing now, what should be done to the driver of the car, and so on, even though he wishes not to wallow in these thoughts and even though these thoughts interfere with other activities that he values highly and wishes to concentrate on (e.g. preparing a report or doing something for another member of the family). One feature of these characteristically human types of emotional states is that there is a partial loss of control of thought processes. This makes sense only in the context where one can sometimes be in control. I call these states perturbant. My claim is that not all information processing architectures support such a context. For example, the purely reactive architecture does not. However, where there is a deliberative layer and a meta-management layer in the architecture it is possible to have "meta-management" goals regarding how the (resource limited) deliberative layer is deployed and to find that those goals are consistently being violated by mechanisms that grab and divert attention away from the chosen tasks. To summarise: sometimes we control thoughts e.g. deciding to cease dwelling on yesterday's humiliating experience and instead pay attention to an important task. But sometimes we fail. So we also have the ability to lose control of our thought processes, at least partly. It might be thought that this is a kind of malfunction which is due to a design flaw, and that a different sort of design could prevent such states occurring. My conjecture is that this is not so. There will have to be mechanisms that can sometimes override high level decisions, because deliberative and meta-management mechanisms are inherently resource limited so that various sorts of sub-processes will need to compete for attention, and it is not safe to suppress all interrupts by always allowing high level decisions to dominate processing. These interrupting mechanisms, however, will sometimes have to act quickly and on the basis of partial information, so they will have the potential for getting things wrong, and causing perturbant states that override top level decisions. It may be possible to suppress these mechanisms in special cases: and that may be what has happened in people whom we regard as completely cold and calculating. CONCLUSION The human brain is perhaps the most complex system ever studied by science and it is not to be expected that we shall achieve a full understanding in the foreseeable future. If we aim for an architecture-based understanding, we can get a much deeper understanding of what sort of control system a human mind is, and how different minds, human and non-human, normal and abnormal, infant and adult, human and machine, are similar, and how they differ. Within this framework we can generate architecture-based concepts for describing classes of mental states and processes that are possible in different architectures, just as the theory of the architecture of physical matter generated a new system of concepts for classifying kinds of stuff, the periodic table of the elements. We still await a periodic table for human minds. Within this design-based approach, enhanced by neurophysiological studies of the underlying mechanisms, we can hope to find deeper theories of how a human mind normally works and also how it might go wrong: the more complex the architecture, the more ways it can go wrong. By applying all these ideas, we should be able to help therapy, counselling and education. Most educational theories are not based on a deep understanding of what kind of mechanism we are talking about, just a lot of hunches and rules of thumb (some of which work very well). So within the framework of AI, in collaboration with other disciplines, we can hope for new insights into matters that concern us all. As yet there is no evidence that this requires a new extension to quantum physics as Penrose claims. Even if we already had such a theory, there is no evidence that it would help us understand the design issues described above any better. ======================================================================= REFERENCES [Beaudoin94] L.P. Beaudoin, Goal processing in autonomous agents, PhD Thesis, School of Computer Science, The University of Birmingham, 1994, available in three formats in the following compressed files in the above ftp directory: Luc.Beaudoin_thesis.ps.Z (postscript) Luc.Beaudoin_thesis.rtf.Z (rtf) Luc.Beaudoin_thesis.txt.Z (plain text, without diagrams) [Sloman92] A. Sloman, `The emperor's real mind: review of Roger Penrose's {\em The Emperor's new Mind: Concerning Computers Minds and the Laws of Physics,'} in {\em Artificial Intelligence,} 56 (1992) pp 355-396 [Sloman93] A. Sloman, `The mind as a control system', in {\em Philosophy and the Cognitive Sciences,} (eds) C. Hookway and D. Peterson, Cambridge University Press, (1993) pp 69-110, [Wright96] I.P. Wright, A. Sloman and L.P. Beaudoin, Towards a Design-Based Analysis of Emotional Episodes, (To appear in) {\em Philosophy Psychiatry and Psychology}, 1996. Many of the papers produced by the Birmingham Cognition and Affect project can be found on the internet in the project directory: ftp://ftp.cs.bham.ac.uk/pub/groups/cog_affect The toolkit we are developing for exploring a variety of agent architectures is described in this online WWW document, with some simple demonstration "movies": http://www.cs.bham.ac.uk/~axs/cog_affect/sim_agent.html [end]