Standalone/Deep Learning
MLP Notation and Trainable Parameters
Dec 26, 2024·5 min read
In this short article, I’ll explain the standard notation used in Multilayer Perceptrons (MLPs), which is crucial for anyone starting with neural networks. You’ll learn how to:
Understand MLP Notation: Get familiar with the basic symbols and terms.
Calculate Trainable Parameters: Determine the number of weights and biases you can train.
Download PDF Notes: At the end, download notes to review anytime.
Multilayer Perceptrons (MLPs) are a fundamental type of neural network used for a wide range of machine learning tasks. Understanding the notation that defines these networks is very crucial for both creating and implementing them effectively. In this part of our discussion, I’ll break down the notation. First, let’s understand the three types of layers in brief.
Layers in an MLP
Input Layer: Comprises neurons that receive inputs directly from your dataset. Each neuron corresponds to one feature.
Hidden Layers: One or more layers that apply transformations to the inputs using weights, biases, and activation functions. These layers are where most of the computation occurs.
Output Layer: The final layer that outputs the prediction or classification result from the MLP.
We begin with data that is four-dimensional, meaning each sample (row) in the dataset contains four distinct features. For any given row, the inputs to the MLP can be specified as Xᵢ₁ , Xᵢ₂ , Xᵢ₃ , Xᵢ₄. Each subscript represents an input feature from one sample (row) in the dataset, where i ranges from 1 to m.
The resulting Multilayer Perceptron (MLP) is shown in the image below, but weights, biases, and outputs are not yet labeled. We will learn about each of these step by step.
Bias Notation
A bias is an additional parameter associated with each neuron and is added to the weighted sum of the inputs and the previous layer’s outputs before applying the activation function. In MLP notation, the bias associated with a particular neuron is typically represented by b, with subscripts to indicate its position within the network. The standard notation used is bᵢⱼ
Where:
i represents the layer number (starting from 1 for the first hidden layer since input layers do not have biases).
j indicates the index of the neuron within layer i.
For example, b₂₁ refers to the bias term for the first neuron in the second hidden layer of the network. Similarly, b₁₁ would be the bias for the first neuron in the first hidden layer. Refer to the image below to view the bias terms for each of the six neurons in the network.
Output Notation
An output is the result produced by each neuron after the activation function processes the weighted sum of the inputs and the bias associated with that neuron. In MLP notation, the output associated with a specific neuron is denoted by O, with subscripts to indicate its position within the network. The standard notation used is Oᵢⱼ
Where:
i represents the layer number (starting from 1 for the first hidden layer, as input layers do not produce outputs in this context).
j indicates the index of the neuron within layer i.
For example, O₂₁ refers to the output of the first neuron in the second hidden layer of the network. Similarly, O₁₁ would be the output for the first neuron in the first hidden layer.
If you observe, the subscript for the output of any neuron Oᵢⱼ is similar to the subscript of the bias of that neuron bᵢⱼ. This consistency in notation helps clarify that each neuron’s output and bias are directly related, simplifying the tracking of data flow and computations across the network. Refer to the image below to see the output notations for each neuron across the layers of the network.
Weight Notation
In a Multilayer Perceptron (MLP), weights represent the strength and direction of the connection between two neurons in consecutive layers. Each connection between neurons carries a weight that determines how much influence the output of one neuron will have on the input of another neuron in the next layer. In MLP notation, the weight associated with a connection between two neurons is denoted by :
Where:
k: Layer number where the weight is applied, specifically from layer k-1 to layer k.
i: Index of the neuron in the previous layer k-1 where the connection originates.
j: Index of the neuron in the current layer k where the connection is targeted.
For example, let us determine the weight of the connection originating from neuron 1 of hidden layer 1 and ending at neuron 2 of hidden layer 2. Here, the value of k is 2 because it represents the current layer (hidden layer #2) where the weight is applied. The value of i is 1 because it refers to the index of the neuron in the previous layer (hidden layer #1) from which the connection originates. The value of j is 2 because it refers to the index of the neuron in the current layer (hidden layer #2) where the connection is targeted. Refer to the image below to see how the weights are labeled between the layers of the network.
Calculating Trainable Parameters
In a Multilayer Perceptron (MLP), trainable parameters refer to all the weights and biases that the network learns during the training process. These parameters are adjusted iteratively to minimize the error and improve the network’s predictions.
What are Trainable Parameters?
Weights: Represent the strength of connections between neurons in consecutive layers.
Biases: Adjust the output of each neuron, allowing the activation function to shift for better performance.
For each layer in an MLP:
Number of Weights: (Number of neurons in the previous layer) × (Number of neurons in the current layer)
Number of Biases: (Number of neurons in the current layer)
The total number of trainable parameters in an MLP is the sum of all weights and biases across all layers. Refer to the image below for a visual representation of how trainable parameters are calculated across layers in an MLP.
Key Takeaways
Weights depend on connections between neurons across layers.
Biases depend on the number of neurons in each layer.
Trainable Parameters = Total Weights + Total Biases
Download the PDF Notes
To make your learning easier, I’ve compiled all the key concepts, diagrams, and explanations from this article into a downloadable PDF file. This resource will serve as a quick reference guide for understanding MLP notations, including inputs, biases, outputs, and trainable parameters.
👉 [Click here to download the PDF notes]
Keep this resource handy while working on your MLP projects or revising key concepts.
Don’t forget to follow me for more such content! 🚀
More in Deep Learning
