Neural networks Self Organizing Map by Engr. Edgar Carrillo II

Neural Networks
Self Organizing Map
By: Edgar Caburatan Carrillo II
Master of Science in Mechanical Engineering
De La Salle University Manila,Philippines

Self Organizing Maps: Fundamentals
1. What is a Self Organizing Map?
2. Topographic Maps
3. Setting up a Self Organizing Map
4. Kohonen Networks
5. Components of Self Organization
6. Overview of the SOM Algorithm

What is a Self Organizing Map?
So far we have looked at networks with supervised training techniques, in which there
is a target output for each input pattern, and the network learns to produce the
required outputs.
We now turn to unsupervised training, in which the networks learn to form their own
classifications of the training data without external help. To do this we have to
assume that class membership is broadly defined by the input patterns sharing
common features, and that the network will be able to identify those features across
the range of input patterns.
One particularly interesting class of unsupervised system is based on competitive
learning, in which the output neurons compete amongst themselves to be activated,
with the result that only one is activated at any one time. This activated neuron is
called a winner-takesall neuron or simply the winning neuron.Such competition can
be induced/implemented by having lateral inhibition connections (negative feedback
paths) between the neurons.The result is that the neurons are forced to organise
themselves. For obvious reasons, such a network is called a Self Organizing Map
(SOM).

Topographic Maps
Neurobiological studies indicate that different sensory inputs (motor, visual,
auditory, etc.) are mapped onto corresponding areas of the cerebral cortex
in an orderly fashion.
This form of map, known as a topographic map, has two important
properties:
1. At each stage of representation, or processing, each piece of incoming
information is kept in its proper context/neighbourhood.
2. Neurons dealing with closely related pieces of information are kept close
together so that they can interact via short synaptic connections.
Our interest is in building artificial topographic maps that learn through self-
organization in a neurobiologically inspired manner.
We shall follow the principle of topographic map formation: “The spatial
location of an output neuron in a topographic map corresponds to a
particular domain or feature drawn from the input space”.

Setting up a Self Organizing Map
The principal goal of an SOM is to transform an incoming signal pattern of
arbitrary dimension into a one or two dimensional discrete map, and to
perform this transformation adaptively in a topologically ordered fashion.
We therefore set up our SOM by placing neurons at the nodes of a one or
two dimensional lattice. Higher dimensional maps are also possible, but not
so common.
The neurons become selectively tuned to various input patterns (stimuli) or
classes of input patterns during the course of the competitive learning. The
locations of the neurons so tuned (i.e. the winning neurons) become ordered
and a meaningful coordinate system for the input features is created on the
lattice. The SOM thus forms the required topographic map of the input
patterns. We can view this as a non-linear generalization of principal
component analysis (PCA).

Organization of the Mapping
We have points x in the input space mapping to points I(x) in the output space:
Each point I in the output space will map to a corresponding point w(I) in the input space.

Kohonen Networks
We shall concentrate on the particular kind of SOM known as a Kohonen
Network. This SOM has a feed-forward structure with a single computational
layer arranged in rows and columns. Each neuron is fully connected to all the
source nodes in the input layer:
Clearly, a one dimensional map will just have a single row (or a single column)
in the computational layer.

Components of Self Organization
The self-organization process involves four major components:
Initialization: All the connection weights are initialized with small random values.
Competition: For each input pattern, the neurons compute their respective values
of a discriminant function which provides the basis for competition. The
particular neuron with the smallest value of the discriminant function is declared
the winner.
Cooperation: The winning neuron determines the spatial location of a topological
neighbourhood of excited neurons, thereby providing the basis for cooperation
among neighbouring neurons.
Adaptation: The excited neurons decrease their individual values of the
discriminant function in relation to the input pattern through suitable adjustment
of the associated connection weights, such that the response of the winning
neuron to the subsequent application of a similar input pattern is enhanced.

The Competitive Process
If the input space is D dimensional (i.e. there are D input units) we can write the input
patterns as x = {x : i = 1, …, D} and the connection weights between the input units i
and the neurons j in the computation layer can be written wj= {wji: j = 1, …, N; i = 1, …,
D} where N is the total number of neurons.
We can then define our discriminant function to be the squared Euclidean distance
between the input vector x and the weight vector wj for each neuron j
In other words, the neuron whose weight vector comes closest to the input vector (i.e. is
most similar to it) is declared the winner. In this way the continuous input space can be
mapped to the discrete output space of neurons by a simple process of competition
between the neurons.

The Cooperative Process
In neurobiological studies we find that there is lateral interaction within a set of excited
neurons. When one neuron fires, its closest neighbours tend to get excited more than
those further away. There is a topological neighbourhood that decays with distance.
We want to define a similar topological neighbourhood for the neurons in our SOM. If S
is the lateral distance between neurons i and j on the grid of neurons, we take
as our topological neighbourhood, where I(x) is the index of the winning neuron. This
has several important properties: it is maximal at the winning neuron, it is symmetrical
about that neuron, it decreases monotonically to zero as the distance goes to infinity,
and it is translation invariant (i.e. independent of the location of the winning neuron)
A special feature of the SOM is that the size s of the neighbourhood needs to decrease
with time. A popular time dependence is an exponential decay:

The Adaptive Process
Clearly our SOM must involve some kind of adaptive, or learning, process by which the
outputs become self-organised and the feature map between inputs and outputs is
formed.
The point of the topographic neighbourhood is that not only the winning neuron gets its
weights updated, but its neighbours will have their weights updated as well, although by
not as much as the winner itself. In practice, the appropriate weight update equation is
in which we have a time (epoch) t dependent learning rate
and the updates are applied for all the training patterns x over many epochs.
The effect of each learning weight update is to move the weight vectors w of the
winning neuron and its neighbours towards the input vector x. Repeated presentations
of the training data thus leads to topological ordering.

Ordering and Convergence
Provided the parameters (sOrdering are selected properly, we can start
from an initial state of complete disorder, and the SOM algorithm will
gradually lead to an organized representation of activation patterns drawn
from the input space. (However, it is possible to end up in a metastable state
in which the feature map has a topological defect.)
There are two identifiable phases of this adaptive process:
1. Ordering or self-organizing phase – during which the topological ordering
of the weight vectors takes place. Typically this will take as many as 1000
iterations of the SOM algorithm, and careful consideration needs to be given
to the choice of neighbourhood and learning rate parameters.
2.Convergence phase – during which the feature map is fine tuned and
comes to provide an accurate statistical quantification of the input space.
Typically the number of iterations in this phase will be at least 500 times the
number of neurons in the network, and again the parameters must be chosen
carefully.

Visualizing the Self Organization Process
Suppose we have four data points
(crosses) in our continuous 2D input
space, and want to map this onto four
points in a discrete 1D output space. The
output nodes map to points in the input
space (circles). Random initial weights
start the circles at random positions in the
centre of the input space.
We randomly pick one of the data points
for training (cross in circle). The closest
output point represents the winning
neuron (solid diamond). That winning
neuron is moved towards the data point
by a certain amount, and the two
neighbouring neurons move by smaller
amounts (arrows).

Cont..
Next we randomly pick another data point
for training (cross in circle). The closest
output point gives the new winning neuron
(solid diamond). The winning neuron moves
towards the data point by a certain amount,
and the one neighbouring neuron moves by
a smaller amount (arrows).
We carry on randomly picking data points
for training (cross in circle). Each winning
neuron moves towards the data point by a
certain amount, and its neighbouring
neuron(s) move by smaller amounts
(arrows). Eventually the whole output grid
unravels itself to represent the input space.

Overview of the SOM Algorithm
We have a spatially continuous input space, in which our input vectors live.
The aim is to map from this to a low dimensional spatially discrete output
space, the topology of which is formed by arranging a set of neurons in a grid.
Our SOM provides such a nonlinear transformation called a feature map.
The stages of the SOM algorithm can be summarised as follows:
1.Initialization – Choose random values for the initial weight vectors w
2.Sampling – Draw a sample training input vector x from the input space.
3.Matching – Find the winning neuron I(x) with weight vector closest to input
vector.
4.Updating – Apply the weight update equation
5.Continuation – keep returning to step 2 until the feature map stops changing.

Application: The Phonetic Typewriter
Patent US3265814
One of the earliest and well known applications of the SOM is the phonetic
typewriter of Kohonen. It is set in the field of speech recognition, and the
problem is to classify phonemes in real time so that they could be used to drive
a typewriter from dictation.
The real speech signals obviously needed pre-processing before being applied
to the SOM. A combination of filtering and Fourier transforming of data
sampled every 9.83 ms from spoken words provided a set of 16 dimensional
spectral vectors. These formed the input space of the SOM, and the output
space was an 8 by 12 grid of nodes.
Though the network was effectively trained on time-sliced speech waveforms,
the output nodes became sensitized to phonemes and the relations between,
because the network inputs were real speech signals which are naturally
clustered around ideal phonemes. As a spoken word is processed, a path
through output space maps out a phonetic transcription of the word. Some
post-processing was required because phonemes are typically 40-400 ms long
and span many time slices, but the system was surprisingly good at producing
sensible strings of phonemes from real speech.

Conclusion
1. We began by defining what we mean by a Self Organizing Map (SOM) and
by a topographic map.
2. We then looked at how to set up a SOM and at the components of self
organisation: competition, cooperation, and adaptation. We saw that the self
organization has two identifiable stages: ordering and convergence.
3. We had an overview of the SOM algorithm and its five stages: initialization,
sampling, matching, updating, and continuation.
4. We ended our discussion with the application of the phonetic typewriter.

References
[1] Haykin: Sections 9.1, 9.2, 9.3, 9.4.
[2] Beale & Jackson: Sections 5.1, 5.2, 5.3, 5.4, 5.5
[3] Gurney: Sections 8.1, 8.2, 8.3
[4] Hertz, Krogh & Palmer: Sections 9.4, 9.5
[5] Callan: Sections 3.1, 3.2, 3.3, 3.4, 3.5.
[6] Mehotra, K., Mohan, C. K., & Ranka, S. (1997). Elements of Artificial Neural
Networks. MIT Press pp. 187-202
[7] Fausett, L. (1994). Fundamentals of Neural Networks. Prentice Hall. pp. 169-187
[8] http://citeseer.ist.psu.edu/104693.html
[9] http://www.cis.hut.fi/research/som-bibl/
[10] Java applet & tutorial information: http://davis.wpi.edu/~matt/courses/soms/

[11] Kohonen T (1995, 1997, 2001) Self-Organizing Maps (1st, 2nd and 3rd
editions). Springer-Verlag, Berlin.
[12] Haykin S (1994) Neural Networks: A Comprehensive Foundation. Macmillian,
New York, NY.
[13] Haykin S (1999) Neural Networks: A Comprehensive Foundation (2nd ed.).
Prentice Hall, Englewood Cliﬀs, NJ.
[14] Oja E, Kaski S (eds.) (1999) Kohonen Maps. Elsevier, Amsterdam, The
Netherlands.
[15] Allinson N, Yin H, Allinson L, Slack J (eds.) (2001) Advances in Self- Organising
Maps. Springer-Verlag, London, UK.
[16] Ritter H, Martinetz T, Schulten K (1992) Neural Computation and Selforganising
Maps: An Introduction. Addison Wesley, Reading, MA.
[17] Van Hulle MM (2000) Faithful Representations and Topographic Maps: From
Distortion to Information Based Self-Organization. Wiley, New York, NY.
[18] T. Graepel, M. Burger, K. Obermayer, (1998) "Self-Organizing Maps:
Generalizations and New Optimization Techniques", Neurocomputing, 20:173-190.
[19] J. Kangas, (1994) "On the Analysis of Pattern Sequences by Self-Organizing
Maps". PhD thesis, Helsinki University of Technology, Espoo, Finland.
[20] Ultsch, A. and Siemon, H. P. (1990) Kohonen’s self-organizing feature maps for
exploratory data analysis. In Proceedings of ICNN’90, International Neural Network
Conference, pages 305-308, Kluwer, Dordrecht.

Neural networks Self Organizing Map by Engr. Edgar Carrillo II

More Related Content

What's hot (18)

Viewers also liked (20)

Similar to Neural networks Self Organizing Map by Engr. Edgar Carrillo II (20)

More from Edgar Carrillo (6)

Recently uploaded (20)

Neural networks Self Organizing Map by Engr. Edgar Carrillo II