Explaining multilayer perceptrons in terms of general matrix multiplication

Having considered An overview of deep learning from a mathematical perspective

and the Significance of non-linearity in machine learning 

we can now explain multilayer perceptrons in terms of general matrix multiplication

A Multi-Layer Perceptron (MLP) is a class of feedforward artificial neural networks (ANNs) that consist of multiple layers of nodes, each fully connected to the nodes in the previous and next layers. 

An MLP typically consists of an input layer, one or more hidden layers, and an output layer. Each layer, except for the input layer, consists of neurons (nodes) that apply a non-linear activation function to the weighted sum of their inputs.

Each connection between nodes in adjacent layers has an associated weight. Each node (neuron) in a layer, except for the input layer, has an associated bias.

We can represent this in terms of matrix multiplication as below

The forward propagation process involves computing the output of each layer using matrix multiplication followed by the application of an activation function.

Activation functions introduce non-linearity into the model, allowing it to learn complex patterns. Common activation functions include ReLU, sigmoid, and tanh.

Thus, we see that the operations in an MLP are fundamentally matrix multiplications followed by the addition of biases and the application of activation functions. By stacking these operations across multiple layers, an MLP can learn to map input features to output targets through training (adjusting weights and biases). In this sense, the primary purpose of the deep neural network is feature extraction or representation learning. In the following posts, we will explain how we can think of convolutional neural networks as an exception to the general multilayer perceptron through matrix multiplication.

Image source: Stanford CS2n course

Equations via chatGPT