Linear algebra for deep learning
- 1. Linear Algebra for Deep Learning
- 2. Introduction • Linear Algebra is a continuous form of mathematics and also central to almost all areas of mathematics like geometry and functional analysis. • In Linear Algebra, data is represented by linear equations, which are presented in the form of matrices and vectors.
- 3. Mathematical Objects Scalar: A scalar is simply a single number. For example 24.
- 4. Vector • A Vector is an ordered array of numbers and can be in a row or a column. A Vector has just a single index, which can point to a specific value within the Vector. For example, V2 refers to the second value within the Vector
- 5. Matrix • A Matrix is an ordered 2D array of numbers and it has two indices. The first one points to the row and the second one to the column. For example, M23 refers to the value in the second row and the third column. A Matrix can have multiple numbers of rows and columns. Note that a Vector is also a Matrix, but with only one row or one column.
- 6. Tensor • A Tensor is an array of numbers, arranged on a regular grid, with a variable number of axes. A Tensor has three indices, where the first one points to the row, the second to the column and the third one to the axis. For example, T232 points to the second row, the third column, and the second axis. This refers to the value 0 in the right Tensor in the graphic below:
- 7. Contd… • Tensor is the most general term for all of these concepts above because a Tensor is a multidimensional array and it can be a Vector and a Matrix, depending on the number of indices it has. For example, a first-order Tensor would be a Vector (1 index). A second- order Tensor is a Matrix (2 indices) and third-order Tensors (3 indices) and higher are called Higher-Order Tensors (3 or more indices).
- 8. Computational Rules 1. Matrix-Scalar Operations • If you multiply, divide, subtract, or add a Scalar to a Matrix, you do so with every element of the Matrix. The image below illustrates this perfectly for multiplication:
- 9. 2. Matrix-Vector Multiplication • Multiplying a Matrix by a Vector can be thought of as multiplying each row of the Matrix by the column of the Vector. The output will be a Vector that has the same number of rows as the Matrix. The image below shows how this works:
- 10. 3. Matrix-Matrix Addition and Subtraction • Matrix-Matrix Addition and Subtraction is fairly easy and straightforward. The requirement is that the matrices have the same dimensions and the result is a Matrix that has also the same dimensions. You just add or subtract each value of the first Matrix with its corresponding value in the second Matrix. See below:
- 11. 4. Matrix-Matrix Multiplication • Multiplying two Matrices together isn’t that hard either if you know how to multiply a Matrix by a Vector. Note that you can only multiply Matrices together if the number of the first Matrix’s columns matches the number of the second Matrix’s rows. The result will be a Matrix with the same number of rows as the first Matrix and the same number of columns as the second Matrix.
- 12. Matrix Multiplication Properties • Matrix Multiplication has several properties that allow us to bundle a lot of computation into one Matrix multiplication. We will discuss them one by one below. We will start by explaining these concepts with Scalars and then with Matrices because this will give you a better understanding of the process.
- 13. 1. Not Commutative • Scalar Multiplication is commutative but Matrix Multiplication is not. This means that when we are multiplying Scalars, 7*3 is the same as 3*7. But when we multiply Matrices by each other, A*B isn’t the same as B*A. 2. Associative • Scalar and Matrix Multiplication are both associative. This means that the Scalar multiplication 3(5*3) is the same as (3*5)3 and that the Matrix multiplication A(B*C) is the same as (A*B)C. 3. Distributive • Scalar and Matrix Multiplication are also both distributive. This means that 3(5 + 3) is the same as 3*5 + 3*3 and that A(B+C) is the same as A*B + A*C.
- 14. 4. Identity Matrix • The Identity Matrix is a special kind of Matrix but first, we need to define what an Identity is. The number 1 is an Identity because everything you multiply with 1 is equal to itself. Therefore every Matrix that is multiplied by an Identity Matrix is equal to itself. For example, Matrix A times its Identity-Matrix is equal to A. • You can spot an Identity Matrix by the fact that it has ones along its diagonals and that every other value is zero. It is also a “squared matrix,” meaning that its number of rows matches its number of columns.