Fundamentals of Neural Networks

NEURAL NETWORKS
Gagan Deep
Rozy Computech Services
3rd Gate, Kurukshetra-136119
rozygag@yahoo.com, 9416011599
1ANN by Gagan Deep, rozygag@yahoo.com

Artificial Neural Network(ANN)
Artificial + Neural + Network

Artificial
 Made or produced by human beings rather than
occurring naturally, especially as a copy of
something natural.
 However, artificiality does not necessarily have a
negative connotation, as it may also reflect the
ability of humans to replicate forms or functions
arising in nature, as with an artificial heart or
artificial intelligence.
 Intelligence expert Herbert A. Simon observes
that "some artificial things are imitations of
things in nature, and the imitation may use
either the same basic materials as those in the
natural object or quite different materials.

Artificial Intelligence
 Artificial intelligence (AI) is the intelligence
exhibited by machines or software. It is an
academic field of study which studies the
goal of creating intelligence.
 The central problems (or goals) of AI research
include reasoning, knowledge, planning,
learning, natural language processing
(communication), perception and the ability
to move and manipulate objects.

Knowledge Based System
 Knowledge-based system is a program that
acquires, represents and uses knowledge for
a specific purpose.
 Consists of a knowledge-base and an
inference engine.
 Knowledge is stored in the knowledge-base
while control strategies reside in the separate
inference engine.

Knowledge-Base
Inference
Engine

Stages of Biological Neural System
 The neural system of the human body consists of three
stages: receptors, a neural network, and effectors. The
receptors receive the stimuli either internally or from the
external world, then pass the information into the neurons in
a form of electrical impulses. The neural network then
processes the inputs then makes proper decision of outputs.
Finally, the effectors translate electrical impulses from the
neural network into responses to the outside environment.
Figure shows the bidirectional communication between
stages for feedback

Neural
 Neural relating to a nerve or the nervous
system.
 Situated in the region of or on the same side
of the body as the brain and spinal cord.
 It came from the Greek word Neuron.

Neuron
 A neuron also known as a neurone or nerve
cell) is an electrically excitable cell that
processes and transmits information through
electrical and chemical signals.
 These signals between neurons occur via
synapses, specialized connections with other
cells.
 Synapses - a junction between two nerve
cells, consisting of a minute gap across which
impulses pass by diffusion of a
neurotransmitter.

 The human body is made up of trillions of cells.
 Neurons, are specialized to carry "messages"
through an electrochemical process.
 The human brain has approximately 100 billion
neurons.
 Neurons come in many different shapes and
sizes.
 Some of the smallest neurons have cell bodies
that are only 4 microns wide.
 Some of the biggest neurons have cell bodies
that are 100 microns wide. (Remember that 1
micron is equal to one thousandth of a
millimeter!).

Neurons vs. Other Cells
Similarities with other cells:
 Neurons are surrounded by a cell membrane
that protects the cell.
 Neurons and other body cells both contain a
nucleus that holds genetic information.
 Neurons carry out basic cellular processes
such as protein synthesis and energy
production.

However, neurons differ from other cells in the
body because:
 Neurons have specialize cell parts called
dendrites and axons. Dendrites bring
electrical signals to the cell body and axons
take information away from the cell body.
 Neurons communicate with each other
through an electrochemical process.
 Neurons contain some specialized structures
(for example, synapses) and chemicals (for
example, neurotransmitters).

The Structure of a Neuron
 There are three basic parts of a neuron: the
dendrites, the cell body and the axon.
 However, all neurons vary somewhat in size,
shape, and characteristics depending on the
function and role of the neuron.
 Some neurons have few dendritic branches,
while others are highly branched in order to
receive a great deal of information.
 Some neurons have short axons, while others
can be quite long. The longest axon in the human
body extends from the bottom of the spine to
the big toe and averages a length of
approximately three feet!

Neuron
One way to classify neurons is by the number of
extensions that extend from the neuron's cell body
(soma).

Bipolar neurons have two processes extending from the cell body (examples:
retinal cells, olfactory epithelium cells).
Pseudounipolar cells (example: dorsal root ganglion cells).Actually, these cells
have 2 axons rather than an axon and dendrite. One axon extends centrally
toward the spinal cord, the other axon extends toward the skin or muscle.
Multipolar neurons have many processes that extend from the cell body.
However, each neuron has only one axon (examples: spinal motor neurons,
pyramidal neurons, Purkinje cells). 16ANN by Gagan Deep, rozygag@yahoo.com

SYNAPSE

Brain Interconnections

BIOLOGICAL (MOTOR) NEURON

Neurons can also be classified by the direction
that they send information.
 Sensory (or afferent) neurons: send
information from sensory receptors (e.g., in
skin, eyes, nose, tongue, ears) TOWARD the
central nervous system.
 Motor (or efferent) neurons: send
information AWAY from the central nervous
system to muscles or glands.
 Interneuron: send information between
sensory neurons and motor neurons. Most
interneuron's are located in the central
nervous system.

Action Potentials
 How do neurons transmit and receive
information? In order for neurons to
communicate, they need to transmit information
both within the neuron and from one neuron to
the next. This process utilizes both electrical
signals as well as chemical messengers.
 The dendrites of neurons receive information
from sensory receptors or other neurons. This
information is then passed down to the cell body
and on to the axon. Once the information as
arrived at the axon, it travels down the length of
the axon in the form of an electrical signal known
as an action potential.

Communication Between Synapses
 Once an electrical impulse has reached the end of an
axon, the information must be transmitted across
the synaptic gap to the dendrites of the adjoining
neuron. In some cases, the electrical signal can
almost instantaneously bridge the gap between the
neurons and continue along its path.
 In other cases, neurotransmitters are needed to send
the information from one neuron to the next.
Neurotransmitters are chemical messengers that are
released from the axon terminals to cross the
synaptic gap and reach the receptor sites of other
neurons. In a process known as reuptake, these
neurotransmitters attach to the receptor site and
are reabsorbed by the neuron to be reused.

Neurotransmitters
 Neurotransmitters are an essential part of our
everyday functioning. While it is not known
exactly how many neurotransmitters exist,
scientists have identified more than 100 of these
chemical messengers.
 The spikes travelling along the axon of the pre-
synaptic neuron trigger the release of
neurotransmitter substances at the synapse.
 The neurotransmitters cause excitation or
inhibition in the dendrite of the post-synaptic
neuron.

 The integration of the excitatory and
inhibitory signals may produce spikes in the
post-synaptic neuron.
 The contribution of the signals depends on
the strength of the synaptic connection.
 What effects do each of these
neurotransmitters have on the body?
 What happens when disease or drugs
interfere with these chemical messengers?
 The following are just a few of the major
neurotransmitters, their known effects, and
disorders they are associated with.

 Acetylcholine: Associated with memory, muscle
contractions, and learning. A lack of
acetylcholine in the brain is associated with
Alzheimer’s disease.
 Endorphins: Associated with emotions and pain
perception. The body releases endorphins in
response to fear or trauma. These chemical
messengers are similar to opiate drugs such as
morphine, but are significantly stronger.
 Dopamine: Associated with thought and
pleasurable feelings. Parkinson’s disease is one
illness associated with deficits in dopamine,
while schizophrenia is strongly linked to
excessive amounts of this chemical messenger.

Biological Prototype
● Neuron
- Information gathering (D)
- Information processing (C)
- Information propagation (A / S)
human being: 1012 neurons
electricity in mV range
speed: 120 m / s
cell body (C)
dendrite (D)nucleus
axon (A)
synapse (S)

Artificial Neural Network
 An Artificial Neural Network (ANN) is an
information processing paradigm that is inspired by
the way biological nervous systems, such as the
brain, process information.T
 The key element of this paradigm is the novel
structure of the information processing system.
 It is composed of a large number of highly
interconnected processing elements (neurones)
working in unison to solve specific problems.
 ANNs, like people, learn by example.
 An ANN is configured for a specific application, such
as pattern recognition or data classification,
through a learning process.

 Learning in biological systems involves adjustments
to the synaptic connections that exist between the
neurones.This is true of ANNs as well.

BRAIN COMPUTATION
 The human brain contains about 10
billion nerve cells, or neurons. On
average, each neuron is connected to
other neurons through approximately
10,000 synapses.

DEFINITION OF NEURAL NETWORKS
 According to the DARPA Neural Network Study
 • ... a neural network is a system composed of many
simple processing elements operating in parallel whose
function is determined by network structure, connection
strengths, and the processing performed at computing
elements or nodes.
 According to Haykin
 A neural network is a massively parallel distributed
processor that has a natural propensity for storing
experiential knowledge and making it available for use. It
resembles the brain in two respects:
• Knowledge is acquired by the network through a learning process.
• Interneuron connection strengths known as synaptic weights are
used to store the knowledge.

NEURAL NETWORKS v/s CONVENTIONAL COMPUTERS
COMPUTERS
 Algorithmic approach
 They are necessarily
programmed
 Work on predefined
set of instructions
 Operations are
predictable
ANN
 Learning approach
 Not programmed for
specific tasks
 Used in decision
making
 Operation is
unpredictable

ARTIFICIAL NEURAL NETWORKS
 Information-processing system.
 Neurons process the information.
 The signals are transmitted by means of
connection links.
 The links possess an associated weight.
 The output signal is obtained by applying
activations to the net input.

ARTIFICIAL NEURAL NETWORKS
 The figure shows a simple artificial neural net
with two input neurons (X1, X2) and one
output neuron (Y). The inter connected
weights are given byW1 and W2.
X2
X1
W2
W1
Y

ASSOCIATION OF BIOLOGICAL NET
WITH ARTIFICIAL NET

PROCESSING OF AN ARTIFICIAL NETWORKS
The neuron is the basic information processing unit of a NN. It
consists of:
1. A set of links, describing the neuron inputs, with weights
W1,W2, …,Wm.
2. An adder function (linear combiner) for computing the
weighted sum of the inputs (real numbers):
3. Activation function for limiting the amplitude of the neuron
output.
j
j
jXWu
m
1


)(uy b

MOTIVATION FOR NEURAL NET
 Scientists are challenged to use machines more
effectively for tasks currently solved by humans.
 Symbolic rules don't reflect processes actually used
by humans.
 Traditional computing excels in many areas, but not
in others.

The major areas being:
 Massive parallelism
 Distributed representation and computation
 Learning ability
 Generalization ability
 Adaptivity
 Inherent contextual information processing
 Fault tolerance
 Low energy consumption

Characteristics of Artificial Neural Networks
 A large number of very simple processing
neuron-like processing elements
 A large number of weighted connections
between the elements
 Distributed representation of knowledge over
the connections
 Knowledge is acquired by network through a
learning process

 The good news: They exhibit some brain-like
behaviors that are difficult to program
directly like:
 learning
 association
 categorization
 generalization
 feature extraction
 optimization
 noise immunity
 The bad news: neural nets are
 black boxes
 difficult to train in some cases

 The NN exhibit mapping capabilities that is they can
input patterns to their associated output patterns.
 The NN learn by example. Thus, NN architecture can
be trained with known examples of a problem before
they are tested for their ‘inference’ capability on
unknown instances of the problem. They can,
therefore identify new objects previously untrained.
 The NN possess the capability to generalize. Thus,
they can predict new outcomes from past trends.
 The NNs are robust systems and are fault tolerant.
They can, therefore, recall full patterns from
incomplete, partial or noisy patterns.
 The NNs can process information in parallel, at high
speed, and in a distributed manner.

Features of Biological Neural Networks
 Some attractive features of the biological NN that
make it superior to even the most sophisticated AI
computer system pattern recognition tasks are the
following
 Robustness and fault tolerance : The decay of nerve
cells does not seem to affect the performance
significantly.
 Flexibility : The network automatically adjust to a
new environment without using any programmed
instruction.
 Ability to deal with a variety of data situations : The
network can deal with information that is fuzzy,
probabilistics, noisy and inconsistent.
 Collective computation : The network performs
routinely many operations in parrallel and also given
task in a distributed manner.

Performance Comparison of Computer and Biological
Neural Networks
 Speed: Brain (Slow in processing information) +
Computer (Fast)= ANN (Fast)
 Processing :Sequential (Programs) & Parallel (Brain)=
Parallel Processing
 Size & Complexity : Billion & trillions of neurons and
their interconnections- so they give size and
complexity.
 Storage : Brain (Adaptable) /Computer (Strictly
Replaceable)- In computers overwriting takes place but
in brain according to interconnection strengths they
add.
 Fault Tolerance : Due to Distributed Networks
information can be retrieved after any
crash/destruction
 Control Mechanism : Central nervous system (Brain)/
Computer (Control Unit)

HISTORICAL BACKGROUND
The history of neural networks that was described
above can be divided into several periods:
First Attempts: There were some initial simulations
using formal logic. McCulloch and Pitts (1943)
developed models of neural networks based on their
understanding of neurology. These models made
several assumptions about how neurons worked. Their
networks were based on simple neurons which were
considered to be binary devices with fixed thresholds.
The results of their model were simple logic functions
such as "a or b" and "a and b".

 Another attempt was by using computer
simulations. Two groups (Farley and Clark, 1954;
Rochester, Holland, Haibit and Duda, 1956). The
first group (IBM researchers) maintained closed
contact with neuroscientists at McGill University.
So whenever their models did not work, they
consulted the neuroscientists. This interaction
established a multidisciplinary trend which
continues to the present day.

 Promising & Emerging Technology: Not
only was neuroscience influential in the
development of neural networks, but
psychologists and engineers also contributed
to the progress of neural network
simulations.
 Rosenblatt (1958) stirred considerable
interest and activity in the field when he
designed and developed the Perceptron. The
Perceptron had three layers with the middle
layer known as the association layer. This
system could learn to connect or associate a
given input to a random output unit.

 Another system was the ADALINE (ADAptive
LInear Element) which was developed in 1960
by Widrow and Hoff (of Stanford University).
The ADALINE was an analogue electronic
device made from simple components. The
method used for learning was different to
that of the Perceptron, it employed the
Least-Mean-Squares (LMS) learning rule.

 Period of Frustration & Disrepute: In 1969
Minsky and Papert wrote a book in which
they generalized the limitations of single
layer Perceptrons to multilayered systems. In
the book they said:
"...our intuitive judgment that the extension
(to multilayer systems) is sterile".
The significant result of their book was to
eliminate funding for research with neural
network simulations. The conclusions
supported the disenchantment of researchers
in the field. As a result, considerable prejudice
against this field was activated.

 Innovation: Although public interest and available
funding were minimal, several researchers
continued working to develop neuromorphically
based computational methods for problems such as
pattern recognition.
During this period several paradigms were
generated which modern work continues to
enhance. Grossberg's (Steve Grossberg and Gail
Carpenter in 1988) influence founded a school of
thought which explores resonating algorithms. They
developed the ART (Adaptive Resonance Theory)
networks based on biologically plausible models.
Anderson and Kohonen developed associative
techniques independent of each other. Klopf (A.
Henry Klopf) in 1972, developed a basis for learning
in artificial neurons based on a biological principle
for neuronal learning called heterostasis.

 Werbos (Paul Werbos 1974) developed and used the
back-propagation learning method, however several
years passed before this approach was popularized.
Back-propagation nets are probably the most well
known and widely applied of the neural networks
today. In essence, the back-propagation net. is a
Perceptron with multiple layers, a different threshold
function in the artificial neuron, and a more robust
and capable learning rule.
Amari (A. Shun-Ichi 1967) was involved with
theoretical developments: he published a paper
which established a mathematical theory for a
learning basis (error-correction method) dealing with
adaptive pattern classification. While Fukushima (F.
Kunihiko) developed a step wise trained multilayered
neural network for interpretation of handwritten
characters. The original network was published in
1975 and was called the Cognitron.

 Re-Emergence: Progress during the late 1970s and
early 1980s was important to the re-emergence on
interest in the neural network field. Several factors
influenced this movement.
 For example, comprehensive books and conferences
provided a forum for people in diverse fields with
specialized technical languages, and the response to
conferences and publications was quite positive. The
news media picked up on the increased activity and
tutorials helped disseminate the technology.
Academic programs appeared and courses were
introduced at most major Universities (in US and
Europe). Attention is now focused on funding levels
throughout Europe, Japan and the US and as this
funding becomes available, several new commercial
with applications in industry and financial
institutions are emerging.
 .

 Today: Significant progress has been made in
the fild of neural networks-enough to attract
a great deal of attention and fund further
research.
Advancement beyond current commercial
applications appears to be possible, and
research is advancing the field on many
fronts.
Neurally based chips are emerging and
applications to complex problems
developing.
Clearly, today is a period of transition for
neural network technology

FEW APPLICATIONS OF NEURAL NETWORKS

We Discussed(marked)
Unit I
 Introduction: Concepts of neural networks,
Characteristics of Neural Networks, Historical
Perspective, and Applications of Neural
Networks.
 Fundamentals of Neural Networks: The
biological prototype, Neuron concept, Single
layer Neural Networks, Multi-Layer Neural
Networks, terminology, Notation and
representation of Neural Networks, Training of
Artificial Neural Networks.
 Representation of perceptron and issues,
perceptron learning and training, Classification,
linear Separability

Thanks!
Gagan Deep
rozygag@yahoo.com
9416011599

Fundamentals of Neural Networks

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Fundamentals of Neural Networks (20)

More from Gagan Deep (7)

Recently uploaded (20)

Fundamentals of Neural Networks