Part I: How the Brain Inspired Machines – The Invention of the Artificial Neural Network (ANN)

Part 1: The Biological Spark – how real neurons inspired ANNs.        

“If you want to understand a really complicated device, like a brain, you should build one.” G Hinton, 2018.

 In 1943, neurophysiologist Warren McCulloch and mathematician Walter Pitts studied and developed many abstract models for neural networks using symbolic logic of Rudolf Carnap and Principia Mathematica. They wrote a paper on how neurons might work, in  order to contemplate how neurons function and behave, they modelled a simple neural network using electrical circuits. Their paper was named as A Logical Calculus of the Ideas Immanent in Nervous Activity . They were the first ones to ever have tried to mimic the behavior of neurons in the human brain. Who knew at that time, that this study would one day bring today’s artificial intelligence revolutionary wave, if only they were here to see it.

Part 2: The Perceptron Era – Rosenblatt's work and early hype.        

In 1958, Frank Rosenblatt created an algorithm for pattern recognition which was known as the perceptron a commendable milestone in AI history. A multilayer perceptron (MLP) comprised 3 layers: an input layer, a hidden layer with randomized weights that did not learn, and an output layer. Later in 1962, he published a book which introduced a proper fully functioning multilayer perceptron which had four layers, in which the last two layers had learned weights. The very same book is considered to be the foundational resource of all deep learning models and systems of today, which is yet again amazing.

In 1967, Shun’ichi Amari published the first deep learning multilayer perceptron trained by stochastic gradient descent. One of his students developed a perceptron with a state of the art architecture where, the system had a five layer MLP with two modifiable layers which were designed and successfully was capable of classifying non-linearly separable patterns. Non-linearly separable classes means, a group of data or information, which cannot be distinguished by the naked eye or any form of visualization, they can only be distinguished by a human through mathematical calculations, and such task being done by a machine was indeed a huge feat achieved along the road to the development of neural network.

Part 3: The AI Winter – setbacks and criticisms        

In 1969, Minsky and Papert famously proved that perceptron could not learn patterns unless they were of a particularly simple kind (i.e. linearly separable). This triggered the beginning of the first neural network winter, during which neural network research was undertaken by only a handful of scientists

Part 4: Hopfield Networks        

“The ability of large collections of neurons to perform “computational” tasks may in part be a spontaneous collective consequence of having a large number of interacting simple neurons”. JJ Hopfield, 1982.

In 1982, JJ Hopfield introduced Hopfield net, which went on to inspire the modern structure of deep learning systems and neural networks. Hopfield provided theoretical framework based on statistical mechanics, which laid the foundations for Ackley, Hinton, and Sejnowki’s Boltzmann machine in 1985. Unlike a Hopfield net, in which the states of all units are specified by the pattern being learned, a Boltzmann machine has a reservoir of hidden units, which can be used to learn complex patterns. The Boltzmann machine is important because it facilitated a conceptual shift away from the idea of a neural network as a passive associative machine towards the view of a neural network as a generative model. the Boltzmann machine demonstrated that neural networks could learn to solve complex problems, implying that, in principle, neural networks could eventually learn to solve almost any problem.

We will explore more facts and insights in the upcoming Part II of this article, in which we will learn more about the history and different studies that led to the eventual invention of the modern structure of ANNs that we use today. Thank you for reading the article up to this point.