SCENARIO: BasicPointing [UNFINISHED] http://www.cs.bham.ac.uk/research/projects/cosy/scenarios/scenario-example1.txt This is an example of the use of the scenario template form http://www.cs.bham.ac.uk/research/projects/cosy/scenarios/scenario-template.txt The example is merely illustrative: if this were a definitive part of a project it would require more detail, and a lot more work would be required to develop and refine the detail. Aaron Sloman Last changed: Tue Mar 22 22:51:27 GMT 2005 TABLE OF CONTENTS -- Scenario label: BasicPointing -- Scenario code: Example1 -- Author(s): Aaron Sloman -- Modified by: -- Scenario summary: -- Motivation: -- Empirical/Scientific context: -- References: -- Precursor scenarios: -- Follow-on scenarios: -- Scenario Ontology: -- RobotOntology: -- Prerequisites: -- -- Reactive mechanisms: -- -- Deliberative mechanisms: -- -- Metamanagement/meta-semantic mechanisms: -- -- Affective states/processes: -- -- Linguistic capabilities: -- -- Kinds of perceptual mechanisms: -- -- Kinds of action mechanisms: -- -- Other prerequisites for the scenario to work. -- Kinds of self-understanding: -- SCENARIO SCRIPTS: -- NEGATIVE SCENARIOS: -- KINDS OF INTEGRATION: -- MODE OF EVALUATION: -- PROGRESS SO FAR: -- -- Design documents -- -- Formal requirements specifications -- -- Code completed -- -- Tests and results of tests -- -- People contributing -- -- Publications -- -- Acknowledgements -- RELATED SITES: ----------------------------------------------------------------------- -- Scenario label: BasicPointing ----------------------------------------------------------------------- -- Scenario code: Example1 ----------------------------------------------------------------------- -- Author(s): Aaron Sloman (Based on discussions with Jeremy Wyatt and others. See http://www.cs.bham.ac.uk/research/projects/cosy/PlayMate-start.html) ----------------------------------------------------------------------- -- Modified by: ----------------------------------------------------------------------- -- Scenario summary: This scenario involves the PlayMate robot showing an ability to point to one or more objects on the table in response to requests. The tasks vary in cognitive difficulty, and therefore in architectural and representational requirements. This is just a first high-level sketch of a family of scenarios which explore different aspects of the ability to point in sequence at a collection of objects. It involves elements of attention insofar as only a subset of objects, or only a subset of features of objects are to be used in selecting items to be pointed at. It also involves attending to more abstract features, such as not having been pointed to previously. In some of the tasks more complex planning operations are involved than in others. A later scenario will address issues of self-understanding that can arise out of this sort of scenario. In some cases the environment does the book-keeping (e.g. preventing repetitions) -- for instance when the objects to be pointed at are aligned in a row, or where instead of merely being pointed at each object is marked, or put in a box. In other cases (e.g. objects scattered around a table) the book-keeping has to be done by cognitive data-structures. The scenario starts with the robot developing procedures (or in some cases using 'innate' procedures) for getting the hand close to a specified object on the table, moving from any arbitrary position. The motion need not be either fast or smooth initially: e.g. a series of slow approximately linear moves may suffice for the initial tasks. (Learning to move smoothly could come later.) This skill can then be used in a variety of tasks involving pointing sequences, including counting for different purposes, matching sets of objects, etc. Sequential operations, external and internal, play an important role in many aspects of cognition. Some (simplified) background theoretical discussion can be found here http://www.cs.bham.ac.uk/research/cogaff/crp/chap8.html ----------------------------------------------------------------------- -- Motivation: The scenario is related to the following goals: 1. Develop competence in controlling the robot arm (specify type of competence) 2. Developing visual competence (specify type of competence) 3. Exploring ways of planning a sequence of moves in a relatively simple context (needs to be elaborated) 4. Modelling some aspects of human competence: - the ability to point in sequence at a set of objects, without omission or repetition of objects. (Probably not shared with many other animals, and not present in very young children) - the ability to react to simple commands by pointing - provides the basis for a later scenario in which the robot understands what it has done and why, and what it did not do and why - could be part of a developmental sequence starting with more primitive perception and motion capabilities This can relate to interesting research issues in developmental psychology, and also in the comparison of capabilities of different animals. (Ref 'clever Hans'). - no attempt is being made here to model specific details of HOW humans and other animals perform such tasks. - can lead to more complex scenarios where pointed at objects are of different sizes - this can also lead to more complex scenarios where the robot has difficulty because some objects are completely hidden, or partly hidden, and where solving the problems require integration of a wider range of reasoning, linguistic, and social competences (e.g. being told by a teacher: 'you left one out'). - counting the same things in different ways, e.g. both individual objects and groups of objects can be pointed at and counted (part of learning about multiplication: 3 groups of 4 = 12). What are the prerequisites for understanding that N groups of K gives the same total as K groups of N? (Requires quite sophisticated meta-management?) ----------------------------------------------------------------------- -- Empirical/Scientific context: Raises questions about how agents learn to cope with more and more difficult conditions for selecting objects attended to or for doing other actions. E.g. Coping with larger numbers of objects. Objects not in a row so that problems of including all objects, and avoiding pointing at the same one twice get harder. Possibility of missing something out because some objects are out of sight Possibility of making a mistake because some objects are moving Downs children have particular difficulties learning to count. Can we shed some light on why? .... more .... ----------------------------------------------------------------------- -- References: Lots of literature on learning to count Check literature on repetitive sequences, pointing actions, etc. in children at various ages http://www.nap.edu/nap-cgi/skimit.cgi?isbn=0309069955&chap=157-180 NAP Skim View of: 5 The Mathematical Knowledge Children Bring to School Identifying perceptually indistinguishable objects: Is that the same one you saw before? John F Santore and Stuart C Shapiro http://www.cse.buffalo.edu/sneps/Bibliography/sansha02.ps (Later papers on the same theme are here in 2004 section) http://www.cse.buffalo.edu/~shapiro/Papers/ Chapter 8 of A.Sloman, The Computer Revolution in Philosophy http://math.about.com/od/counting/ ----------------------------------------------------------------------- -- Precursor scenarios: These include: Simpler scenarios for - perceiving objects and their locations - categorising objects (e.g. by shape, colour, position in some taxonomy) - moving the hand to a perceived object - understanding simpler instructions (e.g. 'point at the red ball') - etc ----------------------------------------------------------------------- -- Follow-on scenarios: Scenarios involving - remembering what one has done (e.g. if asked to point at the objects in reverse order or explaining why a particular pointing sequence was chosen) - being able to answer 'what if' questions about the task - being able to respond to various criticisms (e.g. 'you missed one out') - more capabilities learnt instead of being pre-programmed. ----------------------------------------------------------------------- -- Scenario Ontology: Agents: Playmate and Human PlayMate's hand Other Objects: objects of type: cube, ball, box, table-top shapes: cubic, rectangular block, spherical, flat table top. colours: red, green, blue,... Relations: nearer(x, y, z) x is nearer to y than to z Locations of objects These could be 2-D locations in visual field 2-D locations on table locations identified by relationships (e.g. in the box, between B1 and B2, on top of B3) and many more Trajectory of hand Actions: moving hand to object X (moving it until it is nearer X than any other object) (different levels of chunking) sequences of actions (at different levels of chunking?) Requests: move hand to X point at X point at an x such that P(x) point at (first target, second target, third target) ... point at (all x such that P(x) (more....) ----------------------------------------------------------------------- -- RobotOntology: Robot may know about the whole ontology listed above, or only a subset, in which case there may be some learning that extends the ontology. ----------------------------------------------------------------------- -- Prerequisites: -- -- Reactive mechanisms: Decision to point at A where A is visible and there is nothing in the way should reactively trigger motion of hand until hand is nearer A than any other object. Collisions or near collisions should be detected and reactively cause action to be aborted. (Reactive alarm mechanism?) (E.g. human may move another object while robot is moving its hand, or may move the object the robot is in the process of moving towards.) The simplest case of pointing at all Xs could be a collection of reactive behaviours e.g. point at initially point at leftmost object then repeatedly detect next object to right, until no more objects. -- -- Deliberative mechanisms: When the objects are not in a simple row, the robot may have to plan a sequence. This requires including kinds of knowledge bases, working memories, forms of representation, types of algorithms, etc. Motivational mechanisms, conflict resolution, etc. -- -- Metamanagement/meta-semantic mechanisms: Not required in the basic scenario Would be required for some follow-on scenarios -- -- Affective states/processes: None appropriate in basic scenario. (except perhaps a basic motive to improve performance by noticing opportunities.) In more sophisticated scenarios, surprises could trigger curiosity motive (find why that happened) which triggers learning. -- -- Linguistic capabilities: Understanding speech or text input for requests later scenarios require more linguistic competence, e.g. coping with criticisms and requests for explanations. No linguistic output required in basic scenario. -- -- Kinds of perceptual mechanisms: Different levels of perceptual organisation relevant to parsing the scene, identifying possible trajectories for the arm, identifying objects satisfying descriptions, checking that intended actions are actually performed. -- -- Kinds of action mechanisms: Just moving arm and perhaps aligning fingers. No manipulation (e.g. grasping, pushing) required for basic scenarios. -- -- Other prerequisites for the scenario to work. Humans will have to use only a small range of requests if the robot is to understand. Could be made more versatile later, e.g. 'point at all the objects close to the big box that you did not point at last time' ----------------------------------------------------------------------- -- Kinds of self-understanding: Not a meta-scenario ----------------------------------------------------------------------- -- SCENARIO SCRIPTS: Script 1: Robot is asked to point at something satisfying a description: Point at a box Point at a ball Point at something red Point at a red ball etc. This could be achieved by simply searching the scene for something satisfying the description, selecting it and pointing at it. Script 2: Robot is asked to point at two things: Point at a red box and a blue ball This could be done by selecting the first object to point at then starting again to point at the second. Point at two red balls Robot translates this into Point at a red ball and a red ball Can we find a non-trivial sequence in which the robot first uses the previous strategy then discovers that that can cause it to point at the same thing twice, and that that is not what is required? Somehow it has to learn that the second action has an implicit constraint: don't select the first object. One sort of learning process might simply be trying random ways of forming rules for selecting the second object, until it hits on something like this [X is red] [X is not last object] -> [point at X] Point at all the red balls etc .etc. [details to be filled in] Give an indication of the variability expected. Include possible errors of performance. Kinds of learning or development: Ontological development: Learnt associations: Learnt explanations, models, theories: New formalisms learnt: Modifications/extensions of old formalisms: [to be completed later] ----------------------------------------------------------------------- -- NEGATIVE SCENARIOS: The robot will not be expected to point at very small objects (e.g. peas) or at things like shadows, cracks, parts of objects (e.g. the handle of a cup, or the marks on the surface of an object, etc. The pointing actions need not be fast and fluent, though later scenarios may require this. The robot would not understand point at each red object twice and other more complex pointing, touching, operations ----------------------------------------------------------------------- -- KINDS OF INTEGRATION: Integrates visual perception motor control simple speech/language understanding planning and plan execution grasping of a non-trivial ontology of complex actions composed of simpler ones. .... ----------------------------------------------------------------------- -- MODE OF EVALUATION: (Kind of advance of knowledge demonstrated): E.g. Not a huge amount of science if everything is pre-programmed, but could lead on to investigation of various kinds of learning and how they interact. Could expose some hidden assumptions in psychological theories about how children learn to count. Mostly provides a platform for further developments, e.g. testing meta-management competence where the robot understands what it has and has not done and why, and can relate future performance to past performance: an basis for certain important kinds of learning. demonstration of novel competence (extending the state of the art) [Can other robots do these things?] conformity with empirical evidence of what humans or animals can do [Check psychological literature] conformity with well established theories or explanations [Actually more likely to reveal explanatory gaps.] generation of new specific questions for empirical research, [Several new empirical questions likely to arise out of elaborations of this scenario -- some of them relating to growth of mathematical knowledge.] inspiring new scientific work in other disciplines, e.g. biology, neuroscience, psychology, linguistics, social science? [Maybe investigations of related competences in other animals] where appropriate, passing 'usability' tests [Not sure whether this is relevant. There may be scenarios where domestic help for a disabled person requires a robot to have capabilities built on these primitive capabilities, including counting capabilities, and the ability to answer questions like 'Where is X?' by pointing.] ----------------------------------------------------------------------- -- PROGRESS SO FAR: Only theoretical analysis of scenario -- still incomplete -- -- Design documents -- -- Formal requirements specifications -- -- Code completed -- -- Tests and results of tests -- -- People contributing -- -- Publications -- -- Acknowledgements etc. ----------------------------------------------------------------------- -- RELATED SITES: [To be investigated. ] -----------------------------------------------------------------------