How
hierarchical control self-organizes in artificial adaptive
systems
Rainer W. Paine,
RIKEN Brain Science Institute, Laboratory for Behavior and Dynamic
Cognition, 2-1 Hirosawa, Wako-shi, Saitama, 351-0198,
November, 2004
Abstract
Diverse,
complex, and adaptive animal behaviors are achieved by organizing hierarchically
structured controllers in motor systems. The levels of control progress from
simple spinal reflexes and central pattern generators through to executive
cognitive control in the frontal cortex.
Various types of hierarchical control structures have been introduced and
shown to be effective in past artificial agent models, but few studies have
shown how such structures can self-organize. This study describes how such
hierarchical control may evolve in a simple recurrent neural network model
implemented in a mobile robot.
Topological
constraints on information flow are found to improve system performance by
decreasing interference between different parts of the network. One part becomes responsible for
generating lower behavior primitives while another part evolves top-down
sequencing of the primitives for achieving global goals. A hierarchical neural network is shown
to outperform a comparable single-level network in controlling a mobile robot.
Figure
1 Maze environment. Shows trajectory for a
Left-Right-Right sequence.
A
simulated mobile robot (Khepera II) learns to reach 8 different goals
starting from a home position. Two
types of robot controllers (Figure 2) are evolved using a standard Genetic
Algorithm. The simulations are run
using Webots 4
software. Click HERE for a
demonstration.
Figure
2 The two types of controllers evolved for
the robot in Figure 1. A standard
fully-connected Continuous Time Recurrent Neural Network (CTRNN) with 11 neurons
is shown in (B). Inputs from the robot’s 8 infra-red
sensors are received by 5 neurons.
Two neurons send motor commands to the two wheels of the Khepera
robot. This single network can
encode multiple movement sequences.
The initial activity states of the two “task” neurons determine which
three-turn movement sequence the robot will execute. In the so-called “bottleneck” network
shown in (A), the neural network has
been subdivided into top and bottom levels, such that information flow between
the two levels is limited by the two “bottleneck neurons”. It was found that the bottleneck network
was able to find more goals on average than the standard fully-connected
network. Further, different
functions were automatically adopted by the two levels of the bottleneck
network. The bottom level took over
control of obstacle avoidance (so the robot wouldn’t hit any walls) as well as
left/right turning behavior. The
top level took over control of sequencing the turns in order to find particular
goals. Since obstacle avoidance
requires the robot to respond quickly to changing sensory inputs, the bottom
level neurons evolved to change rapidly to incoming signals, as shown by the
motor output neurons in the bottom of Figure 3. Further, since the overall sequence task
took a much longer time than short-term obstacle avoidance, the task neurons of
the top level evolved to change slowly in response to incoming signals (Figure
3, top).
Figure
3 Neuronal activity for a Right-Left-Right
turn sequence in the bottleneck network.
Top: Neuronal activity of
bottleneck and task neurons with slower evolved time constants to encode turns
and sequences, respectively; Bottom:
Activities of motor output nodes that evolved faster neuronal time constants to
respond to immediate task demands.
Figure
4 Phase space analysis for bottleneck
neuron activities. They determine
turn direction at intersections of the maze in Figure 1. When the activities of the bottleneck
neurons are in the black area, the robot turns left at a corner. When they are in the white area, the
robot turns right. In the grey
area, the robot collides with a wall because the turning behavior encoded by the
bottleneck neurons becomes too strong and overrides the obstacle avoidance
behavior encoded in the bottom level of the network.
A
B
Figure
5 Phase space analysis of task neuron
initial activity states and resulting turn sequences. For example, when the task neuron
initial states are in the orange region at the beginning of movement, then the
robot will make a Right-Left-Right turn sequence, leading it to goal number 6 in
Figure 1. A single neural network
can thus encode multiple movement sequences. The initial states of two neurons, the
so-called “task neurons” in Figure 2, determine which movement sequence the
robot will make. The turn decision
tree structure (A) corresponds to
(B) in which six goals of the maze
in Figure 1 were found. Letters
represent turns of the sequences (L=Left, R=Right).
Figure
6 Left-Right-Left
trajectory of a real Khepera II robot.
The robot controller uses the bottleneck architecture of Figure 2. The controller was first evolved in
simulation and then transferred to the real robot.
Conclusion
This work has
demonstrated an approach to encoding goal-directed, behavioral sequences in a
self-organized recurrent neural network controlling simulated and real mobile
robots. It further examined how
hierarchical segregation of control can emerge in a given architecture and
enhance controller performance.
Different
types of dynamic structures self-organize in different levels of the network for
the purpose of achieving complex navigation tasks. Top-down behavioral switching emerges
through parametric bifurcation of lower level activity via bottleneck
neurons. In the higher level, a
topologically ordered mapping of initial cell activity states to motor-primitive
sequences self-organizes by utilizing the initial sensitivity characteristics of
nonlinear dynamic systems.
Task-neuron modulation
could be effected by even higher level networks which could represent sequences
of sequences for different sets of ever more complex tasks. This research serves as an example of
how complex dynamic structures with initial sensitivity and task-dependent
temporal activity may self-organize to control simpler structures that encode
movement primitives. Such
structures may be analogous to those which encode movement sequences in
biological neural networks, and may be a promising direction for research into
mobile robot navigation.
For further information,
please see my Publications.