School of Computer Science
Personal Web Page


GeRT Logo

GeRT: Generalizing Robot Manipulation Tasks

The Justin robot used in the GeRT project performs one of the example tasks.

In order to work naturally in human environments such as offices and homes, robots will need to be much more flexible and robust in the face of novelty. In this project we are working to develop new methods to cope with novelty in manipulation tasks. Our approach is to enable the robot to autonomously generalize its manipulation skills to new objects. The basic idea is that some successful implementations of a certain robot manipulation task, such as serving a drink, are given as input. These programs then constitute a database of prototypes representing that class of task. When confronted with a novel instance of the same task the robot needs to generalize from the prototypes, establishing appropriate correspondences between objects and actions in the prototypes and their counterparts in the novel scenario. In this way, prototypical task plans may be mapped to a new plan that is suitable for handling different geometric, kinematic, and dynamic task settings, hence solving a task that is physically substantially different but similar at an abstract level. This kind of knowledge transfer or generalization is not restricted just to the most abstract layer. Rather, low-level primitives as well as high-level logical robot actions and operators will be adapted.


The overall aim of the GeRT project is to enable a robot to autonomously generalise its manipulation skills from a set of known objects to previously un-manipulated objects in order to achieve an everyday manipulation task.

Specifically, the project is working towards enabling a robot to generalize from known object instances to previously unseen objects drawn from the same class. In addition we are trying to marry low level robotic control with high level AI planning approaches to enable the robot to reason about how the overall task should influence manipulations of individual objects. The result will be a robot system able to manipulate previously unseen objects to achieve an everyday task, such as making a drink. The research activities in the project will be grouped as follows:

  1. Planning: we are connecting low-level manipulation skills to high level AI planning techniques. This enables the robot to reason about how the overall task affects how it should manipulate a particular object, e.g. if the robot wants to pour liquid from a cup it should not grasp the cup from above.
  2. Learning: we are developing learning algorithms that can adapt grasping strategies so that they are more robust to changes, e.g. the robot should be able to successfully adapt its grasps for one bottle to another.
  3. Perception: we are developing methods for recognizing novel objects as instances of known categories and relating their parts to corresponding parts of known objects. Actions on known objects can hence be transferred to novel objects, e.g. having recognized a handle the robot could adapt an existing grasp for handles.

These research activities are being integrated on the Justin robot system developed at DLR.

The overall approach we are taking in GeRT is to:

  1. Start from existing hand-coded programs as examples;
  2. Learn or adapt the low-level controllers in these programs to handle greater variation in objects;
  3. Automatically create abstract models of action effects from these, which can be reasoned with by a planner;
  4. Use a planner to combine these operators to solve a novel instance of a task in a given domain.

Birmingham's Involvement

Birmingham is primarily involved in steps 2 and 3 of the approach outlined above. This work is divided between two work packages:

Work Package 1: Planning

Using symbolic planning techniques is very desirable for flexibly generating new robot programs adapted to novel tasks. Geometric constraints highly influence the generation process of a robot program. These constraints may be imposed by the robot itself, for example link limits and link geometry, or by the environment, such as objects to manipulate and obstacles. For some changes in the environment, such as a change in object position or the topology of the scene, geometric motion and grasp planners can be used to assist the symbolic planner. However, some task specific geometric relations between manipulated objects, for example the trajectory of pouring water into a glass from a bottle or how the grasp region of the bottle relates to the task, are difficult to formally specify for a geometric planner.

Instead of explicit modelling, these conditions should be learned and generalised from a set of example programs showing the same task and these necessary task relations for a number of objects with different shapes and in the context of different scene arrangements. To simplify the generalisation, the program structure and the sensor data as well as the sensor patterns showing the successful result of an operation will be made available.

Birmingham's involvement with WP1 is led by Dr. Richard Dearden. More information can be found here.

Work Package 3: Learning and Optimisation of Grasp Strategies

Grasping with multifingered robotic hands can be divided into a stage of planning or adapting a grasp and then into the reaching for the intended grasp contacts with the hand. State of the art grasp planners can plan in reasonable time a stable and quite robust grasp for lifting an object. However, the consideration of kinematic constraints of the manipulator (robotic arm) and even more of the functional constraints of the task are quite complicated to address. From example programs grasp preshapes could be used to achieve the same functional constraints while grasping the object. However, it is not likely that the same grasp with same grasp quality could be achieved. Advanced control mechanisms allow implementing compliant reach to grasp movements. Through reinforcement learning and exhaustive testing of different control schemes within a simulation environment robust and almost optimal reach to grasp strategies will be identified for different geometric object types. This will allow more robust adaption of grasp preshapes according to the geometric changes of an object identified in WP2. Structures found in this work package might also lead to more robust example programs.

Birmingham's involvement with WP2 is led by Dr. Jeremy Wyatt and Dr. Rustam Stolkin. More information can be found here.