Download The Matrix Calculus You Need For Deep Learning pdf by Terence Parr, Jeremy Howard, This paper is an attempt to explain all the matrix calculus you would like in order to understand the training of deep neural networks. we tend to assume no maths knowledge beyond what you learned in calculus 1, and provide links to assist you refresh the mandatory math where required. Download the pdf from below to explore all topics and start learning.


1 Introduction

2 Review: Scalar derivative rules

3 Introduction to vector calculus and partial derivatives

4 Matrix calculus
4.1 Generalization of the Jacobian
4.2 Derivatives of vector element-wise binary operators
4.3 Derivatives involving scalar expansion
4.4 Vector sum reduction
4.5 The Chain Rules
4.5.1 Single-variable chain rule
4.5.2 Single-variable total-derivative chain rule
4.5.3 Vector chain rule

5 The gradient of neuron activation

6 The gradient of the neural network loss function
6.1 The gradient with respect to the weights
6.2 The derivative with respect to the bias

7 Summary

8 Matrix Calculus Reference
8.1 Gradients and Jacobians
8.2 Element-wise operations on vectors
8.3 Scalar expansion
8.4 Vector reductions
8.5 Chain rules

9 Notation
10 Resources