School of Computer Science THE UNIVERSITY OF BIRMINGHAM CoSy project CogX project

Possibilities (proto-affordances) Between Form and Function
Aaron Sloman
Last updated: 16 Sep 2009
Installed: 9 Sep 2009

'From Form to Function'


I shall discuss the need for an intelligent system, whether it is a robot, or some sort of digital companion equipped with a vision system, to include in its ontology a range of concepts that appear not to have been noticed by most researchers in robotics, vision, and human psychology. These are concepts that lie between (a) concepts of "form", concerned with spatially located objects, object parts, features, and relationships and (b) concepts of affordances and functions, concerned with how things in the environment make possible or constrain actions that are possible for a perceiver and which can support or hinder the goals of the perceiver.

Those intermediate concepts are concerned with processes, including both processes that are occurring and processes that can occur, and the causal relationships between physical structures/forms/configurations and the possibilities for and constraints on such processes, independently of whether they are processes involving anyone's actions or goals. These intermediate concepts relate motions and constraints on motion to both geometric and topological structures in the environment and the kinds of 'stuff' of which things are composed, since, for example, rigid, flexible, and fluid stuffs support and constrain different sorts of motions.

They underlie affordance concepts. Attempts to study affordances without taking account of the intermediate concepts are bound to prove shallow and inadequate.


The seminar proposal states:
    Our hypothesis is that the form and shape of objects is a key factor
    deciding upon actions that can be performed with an object.

My hypothesis

    Between (a) the form and shape of objects and (b) their functions and
    associated affordances there is (c) a rich variety of phenomena involving
    processes and possible processes that need to be perceived,
    represented and reasoned about.

Type 1 research on vision: objects

    There is much research, both in psychology and AI/Robotics/Machine Vision
    that focuses on learning about object categories and recognition (and/or
    tracking) of instances of those categories.

Type 2 research on vision: scene structures

    There is also much research (going back to the 1960s and earlier) on
    perception of object and scene structure, including locations,
    orientations, curvature and other properties of objects, object parts and
    surfaces, and their relationships. (That includes the "syntactic" approach
    to vision, inspired by work in linguistics on parsing, etc.)

Type 3 research on vision: affordances

    More recently, many people follow J.J.Gibson in emphasising the role of
    perception of affordances in vision and other perceptual modalities
    (e.g. haptic perception of affordances).

For an individual X to perceive affordances is normally construed as

    (1) X gaining information about which actions are possible for X (positive
    affordances) or hindered (negative affordances), where

    (2) X being capable of performing those actions in some situations, and

    (3) performing those actions being able to contribute to achievement of X's
        goals or prevention of states or occurrences that are undesirable for X.

Type 4 research on vision: processes and process possibilities (proto-affordances)

For some time I have been claiming that all three sorts of research (on objects,
structures and affordances) ignore visual competences that lie between
perception of structure and perception of affordances (between perception of
form and perception of function), and are more general and more fundamental than
either, namely:

(a.1) perception of processes in the environment (whether produced by the
    perceiver, or another agent, or simply by physical forces, such as wind,
    gravity, water pressure, changing weather, or plant behaviours, etc.)

(a.2) where the processes need not be relevant to the perceiver's goals or
    preferences and need not be ones that the perceiver is capable of producing
    or preventing, and

Possibilities of and constraints on processes
(b.1) perception of possibilities for and constraints on processes
     of those kinds, independently of whether they are relevant to the
     perceiver's possible actions, goals, or preferences (i.e. perception
     of possibilities and constraints that are not affordances), and

(b.2) independently of whether the perceiver is capable of producing,
     preventing, or constraining those possible processes.

I am not claiming that any individual can perceive ALL such possibilities and
constraints in the environment. A major form of learning is learning to see NEW
types of possible process and NEW types of constraint on processes.

Some of the concepts involved in perceiving, characterising, and reasoning about
actually occurring and possible processes make use of concepts of 'kinds of
stuff', i.e. different sorts of matter with different dispositional properties,
including different sorts of rigidity, breakability, flexibility, elasticity,
fluidity, stickiness, etc.

I call the possibilities and constraints in the environment that are not
affordances 'proto-affordances' since they are the kinds of primitive modal
facts about the environment that underlie affordances.

Two senses of "modal":

 1. Facts are "modal" in the logician's sense (contrasted with "categorical") if
    they are not about what is or is not the case, but about what can or cannot
    or must or may be the case.

 2. Facts can be logically modal yet "a-modal" in the psychologists' sense, i.e.
    not concerned with any particular sensory or motor modality, though
    possibly relevant to several.

    Amodal concepts in the second sense can be either "somatic", i.e. concerned
    with what's going on inside the agent, or "exosomatic", concerned with what
    is happening in the environment (e.g. 3-D surfaces changing their

    (Theories about mirror neurons seem to me to fail to take account of the requirement
    for this kind of abstraction.)

Vicarious affordances

In some cases where affordances are perceived they are not affordances for the
perceiver, but for some other individual, e.g. a child, or prey, or predator, or
collaborator. I call those 'vicarious affordances'.

Epistemic affordances

Besides processes in which physical structures, properties, and relationships
change there are also processes in which information is acquired, e.g. through
vision, touch, hearing, or other senses. Any particular configuration of
a portion of the world supports and hinders the acquisition of particular sorts
of information. The possibilities and constraints on information acquisition are
labelled epistemic affordances. There are many complex relationships
between proto-affordances, action-affordances and epistemic affordances.

In particular, intelligent agents need to work out which epistemic affordances
are relevant to their goals in order not to waste effort attempting to process
information of the wrong kind, e.g. wasting effort trying to cope with noise and
uncertainty where both can be ignored because there is enough unambiguous
information at a higher level of abstraction [8].

NOTE (added 21:44 BST 10 Sep 2009):

    The above characterisation of X being able to perceive affordances can be
    generalised to include seeing or learning what can and cannot be done by
    individuals similar to X (i.e. with similar action capabilities and similar
    kinds of goals, needs, interests, etc.) who are not necessarily present
    at the time of perception.

    This is very close to perception of 'vicarious affordances', except that the
    relevant other other agent need not be present. It can also be related to
    perceiving affordances that X would have if located in a different place
    relative to the objects with those affordances. That can generate an
    explosive set of possibilities, which can be avoided by abstracting away
    from many of the details of particular actions, e.g. representing a wide
    variety of types of grasping simply in terms of two opposing surfaces
    coming together with an object between them.

    A further generalisation would include abstracting away from various details
    of X, e.g. seeing graspability in terms of a wide range of possible grasping
    devices, not just devices similar to the perceivers' hands, claws etc.
    This shades into perception of proto-affordances (See Type 4 research)

Relevant work

The above ideas were summarised, along with some implications for information
processing architectures, in a paper arising out of a previous Dagstuhl
seminar in Feb 2008[1]. See also [2], [3].

More recent research has been attempting to relate development of the ability to
perceive and reason about possible processes in the environment to the
development of an exo-somatic ontology that includes not only perceivable
processes, structures, surfaces, and their changing relationships in the
environment, but also unperceived but inferred kinds of 'stuff' (kinds of
matter) with unperceived but inferred properties that produce dispositions
influencing proto-affordances, e.g. properties such as rigidity, plasticity,
elasticity of various kinds, and various kinds of flexibility [4].

All of these competences need to be integrated within an information processing
architecture that is able to represent and reason about processes involving
continuously changing spatial relationships, including multi-strand processes.

One of the implications of these ideas is that much discussion of so called
'mirror neurones' is seriously confused (as some others have also begun to
realise, e.g. [5], though the arguments are not yet presented very clearly

My presentation will attempt to show that a failure to take account of all of
these logically modal, but psychologically/neurally amodal phenomena between
form and function, between structures and affordances, in an integrated theory,
will cause cognitive science, neuroscience, cognitive robotics, machine vision,
and developmental psychology to flounder about, studying only simplified
fragments of deep and difficult problems.

All this is relevant not only to animal perception and action, but also to
understanding the development of human mathematical competences [6]
and also problem solving in other animals.

REFERENCES [1] @InProceedings {slomanTr0801a, author = {Aaron Sloman}, title = {Architectural and Representational Requirements for Seeing Processes, Proto-affordances and Affordances}, booktitle = {Logic and Probability for Scene Interpretation }, year = {2008}, editor = {Anthony G. Cohn and David C. Hogg and Ralf M{\"o}ller and Bernd Neumann}, number = {08091}, series = {Dagstuhl Seminar Proceedings}, ISSN = {1862-4405}, publisher = {Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, Germany}, address = {Dagstuhl, Germany}, URL = {}, note = {Keywords: Vision, affordances, architectures, development, design space}, } [2] @InCollection {slomanTr0801, author = {Aaron Sloman}, year = {2009}, title = {{Architectural and representational requirements for seeing processes and affordances}}, editor = {Heinke, D. and Mavritsaki, E.}, booktitle = {{Computational Modelling in Behavioural Neuroscience: Closing the gap between neurophysiology and behaviour.}}, publisher = {Psychology Press}, address = {London}, pages = {303--331}, note = {}, } [3] @InCollection {slomanTr0804, author = {Aaron Sloman}, year = {2009}, title = {{Some Requirements for Human-like Robots: Why the recent over-emphasis on embodiment has held up progress}}, editor = {B. Sendhoff and E. Koerner and O. Sporns and H. Ritter and K. Doya}, booktitle = {{Creating Brain-like Intelligence}}, publisher = {Springer-Verlag}, pages = {248--277}, address = {Berlin}, note = {}, } % whole book % [4] Talk 68: Ontologies for baby animals and robots From "baby stuff" to the world of adult science: Developmental AI from a Kantian viewpoint. (PDF) (Last presented at Brown University in June 2009) [5] Pierre Jacob A Philosopher's Reflections on the Discovery of Mirror Neurons Topics in Cognitive Science Vol 1, No 3, July 1009 pp 570--595 [6] Talk 67: Why (and how) did biological evolution produce mathematicians? Alternative title: A New Approach to Philosophy of Mathematics: Design a young explorer, able to discover "toddler theorems" (Or: "The Naive Mathematics Manifesto"). [7] @InProceedings {Sloman96d, author = {A. Sloman}, title = {Actual Possibilities}, editor = {L.C. Aiello and S.C. Shapiro}, booktitle = {Principles of Knowledge Representation and Reasoning: Proceedings of the Fifth International Conference (KR `96)}, publisher = {Morgan Kaufmann Publishers}, address = {Boston, MA}, year = {1996}, pages = {627--638}, isbn = {1-55860-421-9}, note = {}, } [8] @TechReport {slomanDp0702, author = {Aaron Sloman}, title = {{Predicting Affordance Changes (Alternatives ways to deal with uncertainty)}}, month = {Nov}, year = {2007}, note = {Unpublished discussion paper url = {}, number = {COSY-DP-0702}, institution = {School of Computer Science, University of Birmingham}, }

Maintained by Aaron Sloman
School of Computer Science
The University of Birmingham