System Network
Resume:
A Visual Solution to Modular Neural Network System Development
M. Kussul, PhD., A. Riznyk, PhD., E. Sadovaya and A. Sitchov Tie-Qi Chen, PhD.
International Scientific Research Automotive Technologies International, Inc.
Kiev, Ukraine Rochester Hills, Michigan, USA
******@****.****.** ***-**.****@***-*.***
Abstract In this paper, a visual software tool (called
MNN_CAD) for modular neural network system development is Many neural network software packages are commercially
introduced. Its major functionalities such as constructing, available. The most well known include BrainMaker by
training and testing a modular system are described. Its open- California Scientific Software, NeuralWorks by
structure implementation is explained. Finally, the application
NeuralWare, and NeuroSolution by NeuroDimension.
of MNN_CAD in an automotive occupant spatial sensing
These software packages have implemented many popular
problem is presented.
neural network algorithms, and they are widely used by
students, researchers and engineers. However, all these
I. INTRODUCTION
software packages fail to provide an easy yet flexible way of
building modular neural network systems as well as adding
Both modular approach and neural network are well known
new neural network paradigms or other user-customized data
concepts in the research and engineering community. It has
processing modules.
been proven that, by combining the two together, the modular
neural network approach is very effective in searching for
Based on these considerations, we have developed a visual
solutions to complex problems of various fields. Here are a
software tool (called MNN_CAD) for simplifying the process
few examples. The ART (or Adaptive Resonance Theory)
of modular neural network development. MNN_CAD was
networks that simulate human visual cortex are perfect
designed for engineers. With this software, building a
examples of modular neural networks [1][2]. The adaptive
modular neural network system becomes as easy as making a
critic theory for optimization of micro-mechanical factory
2-D CAD drawing. An engineer can build a modular neural
neural control systems is another example [3]. The inverse
network system based on a block diagram in a matter of
kinematics system such as robot arms with joint limits is a
minutes without the need of intensive knowledge of neural
multi-valued and discontinuous function that can be
networks. With MNN_CAD, the engineer can focus more on
approximated by modular neural networks [4]. Other
his/her primary goal to solve the engineering problem,
examples of modular neural networks can be found in short-
which is a tremendous advantage over other conventional
term data prediction [5], variations of committee machines
neural network software tools. For users who do have
[6], etc.
intensive knowledge of the neural networks, MNN_CAD
does provide the means to gain full control to the neural
A modular neural network system consists of several
networks as well as the entire training process.
modules that can be arbitrarily connected to each other. The
modules are not limited to neural networks. They can be data
II. MNN_CAD A VISUAL SOLUTION TO MODULAR
filters, multiplexers, and so on. The dataflow between the
NEURAL NETWORK SYSTEM DEVELOPMENT
modules is triggered by the data itself according to predefined
logics. There are three main reasons why the modular neural
MNN_CAD has two modes: the Design Mode where a
network approach is more effective than the single neural
modular neural network system can be constructed visually
network approach:
and the Train/Test mode where a constructed modular system
1) It is usually possible to simplify a complex task by dividing it
can be trained and tested.
into several smaller subtasks [7]. And, after dividing, it often
becomes easier to select and train a neural network to solve a
specific subtask. A modular system that consists of neural A. Building a modular neural network system
networks trained in this way is a hierarchical mixture of
domain experts [8].
In the Design Mode, one can easily construct a modular
2) It is usually possible to separate independent or less correlated
neural network system using the building blocks provided by
input data sources. The separated data input sources can be
the software. These building blocks are processing modules
processed more efficiently by different neural networks that
with predefined data interfaces. As shown in Figure I (a) and
are trained specifically for them. Moreover, by doing that,
(b), a simple processing module has only input and output
much less data examples will be needed for training the whole
pins, while a more complex processing module may also have
system [9].
3) Compared to single neural network systems, it is usually a switch (so that it can be turned on/off during training) and
easier to improve the desired behavior of modular systems by reference pins (required for supervised learning).
changing the architectures. For example, a modular system
can be made insensitive to the momentary unavailability of
There are two types of processing modules: net (which
certain data input sources, or the distribution of classification
includes various types of neural networks) and block (which
errors can be controlled within desired classes.
includes processing modules other than neural networks). unused data elements correspond to unused input/output pins,
The following neural network modules have been which should be terminated with stubs (see the toolbar shown
implemented in MNN_CAD: multi-layer perceptron with in Figure II (a)).
Delta-Rule (GDR) or Extended Delta-Bar-Delta (EDBD),
Second Order Neural Network (SONN), Random Threshold In order to construct a modular system, one can select
Classifier (RTC), Random Subspace Classifier (RSC), processing modules from the toolbar shown in Figure II (a),
Perceptron-in-Subspaces Classifier (PSC), Principal drag them to the desired location, double-click to change their
Component Analysis (PCA), and Kohonen s Self-Organizing properties, and then connect them with data lines or buses.
Map (SOM). Other implemented processing modules The data lines or buses define the unidirectional data paths
include: normalization, spatial filter for filtering within an between the modules. During processing (i.e. training or
input vector, temporal filter for filtering between consecutive testing), the outputs of previous modules propagate to the
input vectors, multiplexer and switch for data flow control, input nodes of next modules through these paths. A bus
time delay, and data type converter. MNN_CAD also usually contains many data lines. One can connect the data
supports another type of blocks, which are previously created lines within a bus to different destinations by explicitly
modular systems. specifying the line indices. The lines and buses usually
connect different processing modules. However, it is
possible to create a recurrent system by connecting to the
40 10 40 10
same processing module or building a loop that consists of
multiple processing modules.
B1 B2
E 10
(a) A simple processing module (b) A more complex processing
with 40 inputs and 10 outputs. module with 40 inputs, 10
outputs, 10 reference (or target)
outputs for supervised learning,
and an enable/disable switch.
120 5
IN OUT
E 5
(c) A special processing module (d) A special processing module that
that defines the main input defines the output of the modular
source of the modular system. system.
(a) The toolbar for Design Mode. (b) The toolbar for Train/Test
5 6
Mode.
PAT ADD Figure II. MMN_CAD provides easy-to-use toolbars.
After a modular neural network system is drawn,
MNN_CAD automatically checks the design and finds the
(e) A special processing module (f) A special processing module that
possible errors of misuse of the processing modules and the
that defines the reference (or defines the additional input
possible errors in the connections between the processing
target) output of the modular source or alternative reference
modules. Once the design of a modular system passes the
system for supervised learning. output of the modular system.
automatic check, one can start to train or test the modular
Figure I. MMN_CAD provides a collection of ready-to-use
system right away. One can also save the constructed
components (called processing modules), which can be used
modular system as a reusable component for building more
as building blocks to construct any modular system.
complex systems.
Figure I (c), (d), (e) and (f) show the special processing
B. Training and testing a modular neural network system
modules that defines the input and the output of the modular
system. The input (or the reference output) data sources are
Once a valid modular neural network system is
usually associated with data files. However, it is possible to
constructed, one can switch to the Train/Test Mode to train or
use the output (or part of the output) of a neural network
test the modular system immediately.
(called a distal teacher [10]) as the reference (or target)
output of another neural network. It is also possible to use
the data files that have more data elements than needed. The
Depending on how a modular system is constructed, there
are usually many ways to train or test it. One can choose to The C-code generator tool is a necessity if one wants to
train or test a modular system as a whole, and one can choose export the trained modular neural network system. No matter
to train or test one subsystem at a time. A subsystem may how complicated the modular system is, the C-code
consist of one or more neural networks, and a subsystem is generator tool always generates proper source code in
trainable as long as the input and the reference (or target) standard C language so that the trained modular system can
output of the subsystem are available. The most efficient be easily incorporated into any applications on any computer
way to train/test a batch of subsystems is to use queues. Any platforms.
number of connected modules can be added into a queue and
any number of queues can be created. MNN_CAD executes III. OPEN-STRUCTURE IMPLEMENTATION PROVIDES
the queues sequentially. EXTENSIBILITY
MNN_CAD provides different training options. With the MNN_CAD was implemented based on COM (Component
save-best option on, the neural networks are tested Object Model) technology. All the processing modules are
regularly against the independent data during training. Only COM objects with predefined COM interfaces. These COM
the neural networks that have the best classification rates (or objects are DDL (Dynamic-Link Library) files that can be
the least RMS errors) against the independent data are saved. added or replaced without impacting the system. Through the
The draft option is the same as the save-best option predefined COM interfaces, the core unit of MNN_CAD
except that there are fewer checkpoints and the checkpoints interacts seamlessly with all the processing modules that are
are manually preset. This option is useful in finding a present in the system.
reasonably good training depth quickly.
Interface I_Box : IUnknown
Once the training or testing gets started, the statistical // "IUnknown" is the base COM interface.
results of the training or testing process are displayed and {
void Run ;
updated frequently. The statistical results include // Function that performs the main task
// of the processing module to process
input/output distributions, neural network weight
// the data.
distributions, classification distributions, and RMS errors
within epochs, etc. Experiments show that the speed of void Train ;
int Test ;
training or testing of MNN_CAD is much faster than some // Functions that perform training and
commercial software such as NeuralWorks from // testing. These functions are available
NeuralWare. // only in trainable processing modules.
void Init ;
A number of useful tools are available in the Train/Test // Function that initialises the internal
// parameters of the processing module.
Mode. The typical examples of these tools are: module
optimizer, neural network editor and C-code generator . int SetProperty (stProperty *);
stProperty * GetProperty ;
// Functions that provide access to the
The module optimizer tool automatically adjusts the // internal parameters.
internal parameters of a processing module to achieve the char * SetBufferIO (stIO *);
best performance. The internal parameters of a processing // Function that specifies input/output
// of the processing module.
module can be the number of neurons, the number of layers,
the error function, the learning step, or other module specific stStatistics * GetStatistics ;
parameters. One can choose an applicable optimization void StatCollect ;
void StatClear ;
algorithm (such as steepest descent) for the module // Functions that handle the statistical
optimizer . The module optimizer tool can be particularly // results generated during training or
// testing.
useful for certain applications. For example, in an
application where the computational resource is critical, the int Tools (int);
// Function that provides the gateway to
module optimizer tool can help one to find a neural // additional features such as module
network system that has nearly optimal performance and // optimiser and C-code generator.
minimum number of neurons. }
List I. The COM interface of a processing module.
The neural network editor tool allows one to monitor and
change the internal parameters of a neural network module
Presently, only a limited number of neural network
before and during the training process. This tool provides the
paradigms and other processing modules have been
maximum control to a neural network. For example, one can
implemented. However, the open-structure implementation
change the neural network architecture, change any
of MNN_CAD makes it extremely easy to add new
individual weight and transition function, include/exclude
processing modules. In order to interact with the core unit of
any individual neuron from being trained, and so on.
MNN_CAD, a processing module must provide two sets of
interfaces. The first set of interface acknowledges the zone (called ARZ or At-Risk Zone) close to the demarcation
inquiries from the core unit. It tells the core unit what type of line where an airbag deployment with reduced power is
processing module it is and what kinds of functionalities it preferred.
can provide. The second set of interface is used by the core
unit to do the actual work. It can be described by the C++ D is t a n c e i n c r e a s e s i n t h i s d i r e c t i o n .
code shown in List I. Not all the processing modules have to
provide all these functions. For example, the simplest
(Keep-Out Zone)
(At-Risk Zone)
Airbag Unit
processing module may provide only three functions: Run, SSZ
KOZ
ARZ
(S a fe - S e a t in g Z o n e )
GetProperty and SetBufferIO .
D e m a rc a tio n L in e
The source code with such COM interface can be compiled
Figure III. The space in the vehicle passenger compartment can be
into COM objects using compilers like Microsoft Visual C++
divided into three zones.
or Borland C++ Builder. Just copy the compiled DLL files to
the MNN_CAD directory, and immediately the new Channel A (Original Signal)
processing modules will be recognized and loaded, and will
interact with all the existing modules and tools. It is common
200
that a new processing module added by users usually belongs
in one of these three categories: 1) processing modules that
100
perform data conversions between user data formats and the
MNN_CAD internal format; 2) processing modules that
0
0 *-**-**-**-**-** 42
implement new neural network paradigms; 3) processing Channel B (Original Signal)
modules that implement new types of spatial or temporal
filters. 200
IV. APPLICATION IN AN AUTOMOTIVE OCCUPANT
100
SPATIAL SENSING PROBLEM
0
0 *-**-**-**-**-** 42
In this section, to demonstrate the functionalities of the Channel C (Original Signal)
software, MNN_CAD was used in an automotive occupant
spatial sensing problem. There are two reasons why we use
200
this particular example:
1) Engineers who are working on this problem cannot focus
100
on neural network study because they have many other
equally important issues (such as sensors and data
0
collection) to worry about. MNN_CAD enable them to do 0 *-**-**-**-**-** 42
neural network analysis without writing a single line of Channel D (Original Signal)
code. It tremendously reduces the time that is needed for
200
developing a modular neural network system.
2) It must be emphasized that this particular problem is not
like most other classification problems where a success rate
100
of 90% plus is usually considered as good performance.
Because it involves human lives, a success rate of less than
0
0 *-**-**-**-**-** 42
100% will never be considered good enough. Modular
Figure IV. As the occupant subject is moving freely in SSZ, vectors
neural network approach seems a good choice for further are collected from four ultrasonic sensors.
improving the performance even though the single neural
network system already gives high success rate.
Automotive Technologies International, Inc. (Rochester
Hills, MI) has developed an ultrasonic spatial sensing system
A. Problem description
that detects the position of the occupant in the vehicle
passenger compartment. In fact, this system is presently in
When the occupant is in a normal seating position during
production, and has been available on Jaguar XJ8/XJR,
an auto accident, the airbag deployment provides significant
XK8/XKR, and S-Type, beginning in the 2001 model year.
protection. However, when the occupant is near the airbag
The system uses neural networks to process the real-time
unit, it is more likely that the airbag deployment during an
signals from the ultrasonic sensors and then determine in
auto accident will cause harm. Based on this fact, the space
which zone an occupant is at that moment. These neural
in the vehicle passenger compartment can be divided into
networks are trained with 500,000 signal patterns (also called
three zones (see Figure III). The airbag unit should be shut
vectors) that are collected from a matrix of occupant subjects
off when the occupant is in KOZ (or Keep-Out Zone), and it
with different physical sizes and clothing. As an example,
should be turned on when the occupant is in SSZ (or Safe-
Figure IV shows 150 vectors collected from 4 ultrasonic
Seating Zone). Between KOZ and SSZ, there is a middle
sensors as an occupant subject is moving freely in SSZ. The The results of some neural network systems are listed in
4 ultrasonic sensors were installed as follows: 1) sensor A is Table I for comparison. System #1 is a single network
on A-pillar above the instrument panel; 2) sensor B is on B- system that consists of one MLP network. This particular
pillar close to the headrest of the passenger seat; 3) sensor C MLP network was obtained using module optimizer by
is in the center of the roof close to the doom lights; 4) sensor selecting from 120 trained networks. System #2 is a modular
D is on the instrument panel close to the radio/vent controls. system that combines decisions from networks trained with
different decision boundaries. System #3 is a modular system
B. Single neural network vs. modular network system that combines decisions from neural networks trained with
different data inputs. System #4 is the same as System #2
As the initial attempt to solve this classification problem, except that System #4 was optimized using module
two types of neural networks were tried: 1) MLP or multiple optimizer .
layer perceptron, 2) RSC or random subspace classifier [11].
The decision boundaries generated by these two types of The distribution of the classification rates of System #2
neural networks have different shapes in the input data space. against the independent data set is shown in Table II. One
Depending on the characteristics of the data in the input data can see that the classification errors between SSZ and KOZ
space, one shape of decision boundaries usually works better are the smallest ones among all classification errors.
than others. Our experiments show that, for this particular
TABLE I. SUCCESS RATES OF DIFFERENT NETWORK SYSTEMS.
problem, MLP gives better classification rate than RSC (97%
Self Independent Validation
vs. 95%). Network System
Test Test Test
97.45% 98.26% 97.72%
#1
The first modular approach combines decisions from 99.43% 99.11% 98.71%
#2
neural networks that are trained with different decision 99.63% 99.04% 98.85%
#3
99.72% 99.17% 99.17%
boundaries. In this approach, a two-layer structure consisting #4
of seven neural networks is used (see Figure V). The six
TABLE II. DISTRIBUTION OF CLASSIFICATION RATES OF
neural networks at the first layer are trained with different
SYSTEM #2 AGAINST THE INDEPENDENT DATA SET.
decision boundaries. These decision boundaries are: 1) SSZ Target Class SSZ ARZ KOZ
vs. ARZ, 2) ARZ vs. KOZ, 3) KOZ vs. SSZ, 4) SSZ vs. 1.14% 0.09%
Classified As SSZ 99.29%
ARZ+KOZ, 5) ARZ vs. KOZ+SSZ, 6) KOZ vs. SSZ+ARZ. 0.51% 0.24%
Classified As ARZ 98.36%
The neural network at the second layer processes the outputs 0.20% 0.50%
Classified As KOZ 99.67%
from the networks at previous layer and then produces the
There are different ways to train a modular neural network
final output of the system. The major advantage of using this
system. One way to train a modular system with a two-layer
approach is that the occurrence of classification errors can be
structure is that the final network at the second layer is
better controlled. More specifically, the misclassification
trained after all the networks at the first layer have already
errors between SSZ and KOZ can be greatly reduced. Our
been trained separately. Another way is that the modular
experiments show that a fewer number of neural networks
system can be trained as a whole. In this case, all the
can be used at the first layer. In fact, using the four networks
networks in the system learn cooperatively from the errors
that correspond to the decision boundaries 3), 4), 5) and 6)
made by the final network at the second layer. So far, only
produces the best classification rates (99.0 0.5%).
the simplest cooperative learning rule has been implemented
in MNN_CAD: during each cycle, a neural network is trained
The second modular approach combines decisions from
only if both this neural network and the final neural network
neural networks that are trained with different data inputs.
of the modular system produce errors. More complex
This approach also uses a two-layer structure. The neural
cooperative learning rules will be implemented later.
network at the first layer takes different data inputs. If the
data inputs are divided by the channels from which the
Both training methods were investigated and both methods
signals are collected, up to 15 combinations can be used for a
produced similar success rates. However, there is an
four-channel system. The neural network at the second layer
interesting difference in the results produced by these two
processes the outputs from the networks at previous layer and
methods:
then produces the final output of the system. The major
In the system where all the networks are trained separately, all
advantage of using this approach is that the modular system is
the networks at the first layer have very high success rates
usually much more robust to sensor blockage than a single
(97% ~ 99%). These networks will perform well even outside
neural network system. Our experiments show that, by using
the modular system for their subtasks.
three neural networks at the first layer and letting each In the system trained as a whole, the networks at the first layer
network take inputs from three out of the four channels, the usually do not have high success rates (70% ~ 90%). However,
modular system has the best classification rates (98.5 0.5%). the collective contribution of these networks in a modular
system does produce good performance.
C. Results and discussions
Figure V. This is a visual representation of a two-layer modular neural network system for occupant spatial sensing. The six networks (B4 ~ B9)
at the first layer are trained with different decision boundaries.
V. CONCLUSION REFERENCES
[1] Stephen Grossberg, Adaptive pattern classification and universal
We have introduced a visual software tool (MNN_CAD)
recoding II: feedback, expectation, olfaction, and illusions, Biological
for modular neural network system development. The major
Cybernetics, Vol.23, pp.187-202, 1976.
functionalities of MNN_CAD, such as constructing, training, [2] Gail A. Carpenter and Stephen Grossberg, A massively parallel
tuning, and testing a modular system, are described. This architecture for a self-organizing neural pattern recognition machine,
Computer Vision, Graphics, and Image Processing, Vol.37, pp.54-115,
software tool provides great technical capabilities in the form
1987.
of easy user interfaces. It reduces the time needed for the
[3] D. C. Wunsch, N. N. Kussul and M. E. Kussul, Adaptive critic design
process of modular neural network system development to a for optimization of micromechanical factory neural control systems,
matter of minutes. Its open-structure implementation Proceedings of the EUFIT 97, Vol.1, pp.528-533, Aachen, Germany,
1997.
provides the convenience for changing or adding neural
[4] E. Oyama, A. Agah, K. F. MacDorman, T. Maeda and S. Tachi, A
network paradigms and other data processing modules.
modular neural network architecture for inverse kinematics model
learning, Neurocomputing, Vol.38-40, pp.797-805, 2001.
As a demonstration, MNN_CAD was used to process the [5] Yao-Wu Chen, Le-Yu Wang and Hong-Yu Long, Short-term load
forecasting with modular neural networks, Proceedings of the CSEE,
ultrasonic data in an automotive occupant spatial sensing
Vol. 21:4, pp.79-82, 2001.
problem. A number of modular and single neural network
[6] S. Haykin, Neural Networks: A Comprehensive Foundation, Second
systems were constructed, trained, tested, and compared. The Edition, Englewood Cliffs, NJ, Prentice-Hall, 1999.
results show that, by using modular systems, significant [7] Hsin-Chia Fu, Yen-Po Lee, Cheng-Chin Chiang and Hsiao-Tien Pao,
Divide-and-conquer learning and modular perceptron networks, IEEE
improvements can be achieved in terms of classification rate,
Transactions on neural networks, Vol.12:2, pp.250-263, 2001.
generalization capability, as well as system robustness and
[8] M. I. Jordan and R. A. Jacobs, Hierarchical mixture of experts and the
stability. EM algorithm, Neural Computation, Vol. 6, pp.181-214, 1994.
[9] R. D. Reed and R. J. Marks II, Neural Smithing, MIT Press, 1999.
[10]M.I. Jordan and D.E. Rumelhart, Forward models: Supervised learning
Due to limited length of the paper, many details about
with a distal teacher, Cognitive Science, Vol. 16:3, pp.307-354, 1992.
MNN_CAD cannot be presented here. Please feel free to
[11]Ernst Kussul, Dmitri Rachkovskij, and Donald Wunsch, The random
contact the authors for more information and demonstration subspace coarse coding scheme for real-valued vectors, Proceedings of
of the software. IEEE/INNS International Joint Conference on Neural Networks 99,
Washington DC, USA, 1999.
Contact this candidate