From Aaron Sloman Mon Jun 15 10:58:53 BST 1998 To: nigel.birch@epsrc.ac.uk Subject: EPSRC Human Factors Strategy Dear Nigel, Thanks for sending me the strategy document. I am sorry I was not able to comment by the deadline of 12th June: I've been unable to keep up with a host of deadlines in the past month. I hope my hastily written comments a few days late will be of some use. Let me know if you would like to have a nicely formatted printed version. Apologies for length. I hope the tone gives no offence, but fear that's inevitable. I am making these comments available at my web site: http://www.cs.bham.ac.uk/~axs/misc/epsrc_human_factors.text Aaron ======================================================================= EPSRC: IT & COMPUTER SCIENCE PROGRAMME DEVISING A STRATEGY FOR THE SUPPORT OF HUMAN FACTORS: A DISCUSSION PAPER URL http://www.epsrc.ac.uk/progs/technology/it-cs/hfstrcon.htm Comments by Aaron Sloman: CONTENTS (1) Defining the problem space (2) How do people change? (3) Cognitive architectures (4) The relevance of industrial research (5) Human factors and machine intelligence: increasing symmetry ADDITIONAL NOTES: A. A bizarre claim in the document B. The need for responsive mode funding C. This whole field is far too large to be managed on a national scale. D. Support for research infrastructure My main comment is that I felt the document was not based on a proper design analysis but simply reflected a collection of views of groups of researchers who happened to see themselves as working in the field. (I have not read the "People and Computers" booklet -- it may overcome some of the limitations of this document.) What I mean by a design analysis would start from an attempt to characterise in a general and principled way what is currently known about the total situation of a human being interacting with one or more machines (and often simultaneously with other humans) in a social or organisational context. The list of headings at the beginning "People", "Domains of application" and "Technologies led me to expect such a survey or the results of such a survey, but I didn't find it --- perhaps I missed something. Such a principled survey would include the following sorts of analyses: (1) Defining the problem space There are several dimensions of variation in human-machine interaction including the following: (i) There are enormous differences between different kinds of people (some innate, some a result of individual learning, some a result of the culture, some a result of defects such as blindness or other disabilities, some of them motivational, some cognitive, ...). A good theory, and good designs, both need to take account of these differences in a principled way. (ii) There are enormous differences between the purposes for which people interact with machines. This includes both domains of application and what the user is doing in relation to the domain. E.g. controlling a system, learning to control the system, and finding out whether the control facilities are adequate to the domain are quite different purposes even when the domain is the same. (iii) Related to (ii) but partly orthogonal to it, there are enormous differences between the types of machines and modes of interaction they permit. Does anyone have a good theory of the dimensions in which these can vary? To be relevant it needs to be based on a theory of how minds and bodies work, so that only relevant dimensions are included. (This includes dimensions relevant to producing or preventing RSI and dimensions relevant to producing or preventing boredom, incomprehension, etc.) (iv) There are enormous differences in social and/or organisational contexts (home, school, office, emergency service, plane cockpit, station information service, etc.) Many of these impact on motivational and emotional processes. It makes a difference whether how you perform is compared with how others do. Systems designed without a deep understanding of all these dimensions of variation may end up being suitable only for tiny sub-spaces, with huge loss of opportunity as a result. Any general overview of the field should be based at least in part on a survey of such dimensions of variation, and address the great difficulty in covering all combinations in a national research strategy, or even an international strategy. Of course, it should be obvious that what I've called "dimensions" are not dimensions in vector spaces with numerical coordinates. Each "dimension" is itself a complex space. For instance the space of types of minds and the space of types of purposes are not linear spaces and can't even be characterised by vectors: they do not have a homogeneous topology. (Unfortunately people who examine such spaces often restrict themselves to what can be depicted on paper, e.g. 2-D tables and graphs.) Can the total space, combining all these dimensions of variations be divided up in some useful way? A taxonomy of sub-regions in the total space of combinations might provide a good way to think about how to evaluate research (and research proposals) in this field. Some points to note about these dimensions of variation: Regarding (i) the same person can be a different kind of person in different contexts, e.g. an advanced expert in some contexts and a total novice in others. (I am useless at doing anything with a PC.) Are there ways in which transfer of knowledge and skill can be facilitated? See also point (2) below. Regarding points (ii) and (iii) the situation is not static: both tasks and technology are changing rapidly and any general research strategy must include an attempt to get to grips with the changes and the dynamics which drive them. Research in this sort of field can be out of date soon after it starts. (2) How do people change? Related to point (i) above is the fact that people change in various ways for all sorts of different reasons, including natural development, increasing skill with practice, forgetting through lack of practice, conceptual learning, illness, brain damage, senility, cultural changes, changing interests and preferences, etc. In particular, dramatic changes can be brought about by the human+machine interaction itself e.g. through development of new skills, learning of new concepts and ways of thinking, and exposure to new possibilities which trigger motivational changes. Thus one of the major research problems is to understand the variety of processes of change in people (perhaps designers and managers as well as users?) and the laws (if any) defining the space of possible trajectories. This will, of course, be related to general issues in the psychology of learning, development, motivation, perception, etc. Computers are just a subset of machines, along with levers, pencil and paper, packs of playing cards, window catches, etc. (3) Cognitive architectures. One of the most important facts about the total human+machine system is that human beings have motivations and emotions, which make a huge difference to what happens, as every teacher knows (though not all know how to use this knowledge). This fact is not mentioned anywhere. More generally, I get the impression that the contributors to section (e) on cognitive science have a very piecemeal and probably excessively narrow view of what is needed in an adequate theory of what's going on when people interact with machines. This is partly because of the difficulty and breadth of the field, which increasingly has to take account of rapidly growing knowledge in neuroscience as well as developments in many areas of psychology, linguistics, AI, etc. It could be important to understand the information processing architecture of a human being as made up of layers which evolved at different times and are only partly integrated, though there are rich mutual interactions. I also think that currently our knowledge of mechanisms capable of explaining how humans and animals cope with spatial structure and motion is pathetically limited. (Section (b) seems to suggest otherwise.) Even though there's a great deal of factual information available about WHAT such mechanisms can achieve (thousands and thousands of research publications, for instance), I think we know very little about HOW they achieve it, though there are many models of special, limited, cases (e.g. edge detection - contrast the recognition processes of an expert pianist sight reading a Beethoven sonata). Understanding vision is crucial for any theory of how people can interact with machines which increasingly use high bandwidth dynamic visual forms of interaction, both by producing richer and faster moving displays (2-D and 3-D) but also by using TV cameras watching the user (including facial expression, posture, gestures, etc. as well as perhaps the performance of tasks). This is an example of the growing symmetry mentioned below. As virtual reality systems become more sophisticated some of the differences between processes within the machine and external processes will diminish: e.g. machines will need sophisticated 3-D visual capabilities to control simulated agents in simulated environments possibly cohabiting with other agents controlled by, or representing, users. (4) The relevance of industrial research. The document mentions industrial progress on speech as a reason for doubting that academics can compete. If valid, I suspect this argument could be generalised in relation to ALL research areas in human factors. I don't work in speech but I would expect industrial/commercial solutions to be partial and often ad hoc. The latest speech to text systems working on continuous speech are very impressive. What about speech *understanding*? More generally, there is a huge amount of commercial/industrial/military effort going into the design of new sorts of games, control interfaces, decision aids, etc. This covers many areas of human factors. Is there any hope that academic researchers can compete? As with speech it may turn out that the industrial solutions are specialised and partly ad hoc, leaving a need for more academic research to address the deeper, long term, unsolved problems. However, I suspect that far too many of the academics working in this field are trained mainly as psychologists and not as system designers. Consequently they know how to take measurements and how to run statistical packages, and maybe even how to evaluate some systems, but they are often not the right people to think about how to extend the state of the art: that's because psychology degree courses (in this country) do not train people to understand, produce, or debug information processing engines or to understand their modes of interaction. Graduates often don't know the difference between shallow verbal descriptions of surface features of behaviours and deep explanatory theories of the underlying mechanisms. (Of course there are a few exceptions.) So it is very likely that far more progress will come in the near future from the extremely well-funded commercial and industrial laboratories: e.g. those designing computer games or flight simulators, especially if they employ gifted and creative designers rather than psychologists and "human factors" experts. (Many of them have learnt from experience whom to employ.) Unfortunately the research in these commercially motivated labs (or military labs) will not necessarily be published and made generally available (except via products). It may also be too limited, and results may not generalise beyond the specific applications. There may be scope for some academics with a longer view and a broader vision to come up with important new ideas that will not emerge from industrial labs which have to feed into short term profits. Unfortunately I believe the current dynamics of peer review and political influences on selection criteria mean that it is unlikely that such work will be funded. EPSRC should not fund research of a type which is very likely to be done in any case in a commercial laboratories unless it addresses some deep issue or long term problem that is likely to benefit more from an academic context than a commercially driven project. In general, when funds are scarce, I would recommend trying to get industry to fund the academic researchers interested in solving short term practical problems (better interfaces for doing X) and use EPSRC funds only for deeper, broader, more theoretically interesting research. I know this goes against some recent political views. Academics (and scientific administrators) should stand up more strongly against short sighted politicians. (5) Human factors and machine intelligence: increasing symmetry I have often found that people who work on HCI or Human factors believe that the systems they study are inherently asymmetric: with humans on one side and machines on the other, and not much in common. They may acknowledge that humans have complex cognitive mechanisms but they don't think they need to understand the details of those mechanisms (e.g. how perception, learning, problem solving, motivation, attention, etc. work in humans) as long as they measure their effects. I hope it is obvious that this is likely to lead to shallow research. Moreover, it ignores the fact that increasingly machines are being given, and will increasingly NEED to be given, human-like competences if they are to be able to react naturally and fruitfully with human users. The ability to understand, or at least transcribe, speech is an obvious example where machines are making progress, and will need to make a lot more progress if they are to understand the speech. Another, which is not often noticed, is the requirement as graphical interfaces become more sophisticated for machines to have the same kind of grasp of visual structures as the users do. This is especially the case when the structures are created by the user, not generated by the machine from some formula. However, even machine generated images and diagrams may be seen by the user in a way that does not correspond to the structure which generated it. If the machine cannot look at what it has drawn and find the new way of analysing and interpreting it, it may be unable to comprehend a deep misunderstanding and take remedial action. More generally, machines with which we interact will become increasingly autonomous and flexible in ways that require them to have rich cognitive architectures, including capabilities concerned with motivation, learning, planning, finding explanations, creating new designs and new ideas, perceiving things, communicating in various ways, etc. In short the asymmetry of the human+machine interface will diminish. Achieving all this is unlikely to be possible without taking account of how human minds and brains work even if some of the detailed mechanisms are different. Some of the papers in a workshop to be held at AAAI98 on toolkits for designing intelligent agents address this issue. See http://www.cs.bham.ac.uk/~bsl/aaai98/ Alas the workshop attracted only a narrow range of contributors. It might be interesting to find out whether the UK Human factors community can communicate with people thinking about such topics. Unfortunately people are often too concerned with keeping up with their own subfield, to learn about different approaches and perhaps too concerned about dilution or diversion of scarce funds! ADDITIONAL NOTES: A. A bizarre claim in the document Section (b) includes the statement: "There is now a realisation that all sense organs are under the control of the motor system". This does not inspire confidence in the document or the process which produced it. That just happens to be ONE theory. Unless "the motor system" is given an entirely new interpretation, or "control" means "partial influence", I think this is just false. Sometimes it's the other way round: e.g. the visual saccades triggered by detection of a change is an example of the motor system being under the control of a sense organ. (I could enlarge on the underpinnings of many forms of behaviour from startles to sight-reading piano music, etc.) More generally, motivational mechanisms control both sense organs and motor systems. But not totally: perception is partly driven by incoming data and some motor control processes can be highly skilled and autonomous. Control is rarely one-way. One way of reading section (b) suggests that sensory motor systems are not deeply connected with the rest of the architecture (including for instance motivational mechanisms, conceptual knowledge, etc.). If that was intentional then it is surely deeply mistaken. However I agree with the claim that understanding how biological systems (including perhaps toad brains!) work is crucial for this field. Compare my previous points about growing symmetry of the human machine interface. However, it is not enough just to collect data on what the biological systems can and can't do. We need deep explanatory theories. Are there any? B. The need for responsive mode funding My feeling is that it is so difficult to do good research in this field, and so hard to predict what sort of work will turn out to be of long term value that the funding should be entirely responsive, provided that the reviewers and selection committees are encouraged to be both broad minded about new ideas, highly critical of ``more of the same'' and willing to take chances with new researchers who have not yet had a chance to prove themselves. If responsive mode funding produces an imbalance that may not matter if the research gaps are being filled in other countries. (See (C) below.) But we do need mechanisms for reviewing research proposals which are not loaded in favour of what the current community does, as often happens in peer review. It may be better to assess the creativity, intelligence, breadth of knowledge and productivity of grant proposers than the precise content of what they propose to do! I suspect some of the current imbalance is a result of the (real or imagined) political pressure to produce "industrially relevant" research with likely short term commercial benefits. This comes in part from the following attitude expressed in the document: The task is thus to devise a strategy that will: (a) address the imbalance in the research portfolio; and, (b) provide a mechanism for involving users in the process of guiding the strategy, and taking up the results. Commercial organisations that want "results" in this field will do better to produce them themselves rather than depend on over-worked under-resourced academics who in any case are, or should be, thinking about the problems current industry has not yet dreamed of. C. This whole field is far too large to be managed on a national scale. Increasingly work has to be shared or divided across national boundaries. If there are subfields that are better funded and making more progress in other countries there may be no point in trying to compete in the UK: except to ensure that there's a minimal competence here which can feed into educational requirements. This is true of all disciplines, but especially true in a field that is very broad and inherently multi-disciplinary: in fact not a discipline at all. I am not advocating funding of multi-national collaborative projects: I regard these as very wasteful. D. Support for research infrastructure As a result of drastically reduced funding over the last 20 years, and redirection of funds through research councils, it is now very hard for UK university labs to be adequately funded from university funds. The falling cost and increasing power of computers makes it possible now to have access to far more power than ever before. But some of the research needs more than can come from departmental budgets, especially where commercial software packages are needed. It is also hard to get adequate support with installation and integration of software. So there may be a case for encouraging relatively small grant applications to "top-up" departmental equipment provision. Perhaps there should also be a few sites which employ programmers who are available to help and advise researchers in different institutions. It is no longer necessary to be in the same room or even the same building to help someone with software problems, especially if they don't use PCs! (For instance I can often help people in other places using the toolkit I've developed.) This need for funds for infrastructure is not restricted to Human Factors research of course. It may also be useful to provide funds for multi-disciplinary workshops designed to bring together people working on related topics who would normally attend different conferences and workshops or contribute to different journals. However it may be difficult to identify potential contributors with the ability to communicate across boundaries and the time to spare -- since academics now working under such intense pressure. (I feel very sorry for younger colleagues now entering academic careers.) ======================================================================= I hope these hastily written comments are of some use. If I had more time I'd try to find a way to make them more tactful! Name Aaron Sloman, Address School of Computer Science, The University of Birmingham Birmingham B15 2TT Email: A.Sloman@cs.bham.ac.uk URL: http://www.cs.bham.ac.uk/~axs/