7
$\begingroup$

I'm deriving backpropagation step of training neural networks using vectorized equations. Following are two forward propagation equations between last hidden layer and output layer.

$$ Z^{[2]} = W^{[2]}A^{[1]} + b^{[2]} $$ $$ A^{[2]} = \hat{y} = g(Z^{[2]})$$

Where $g(z)$ is activation function.

Now in backpropagation, we calculate the change in cost function ($J$) w.r.t. all the parameters.

I've successfully calculated $\partial{J}/\partial{A^{[2]}}$ and $\partial{J}/\partial{Z^{[2]}}$ using chain rule. Now to calculate $\partial{J}/\partial{W^{[2]}}$, I've formed following chain

$$ \frac{\partial{J}}{\partial{W^{[2]}}} = \frac{\partial{J}}{\partial{A^{[2]}}} \frac{\partial{A^{[2]}}}{\partial{Z^{[2]}}} \frac{\partial{Z^{[2]}}}{\partial{W^{[2]}}}$$

Now to calculate $\frac{\partial{Z^{[2]}}}{\partial{W^{[2]}}}$, I used $ Z^{[2]} = W^{[2]}A^{[1]} + b^{[2]} $ equation which simply gives the derivation $ A^{[1]} $. But in literature, it's given as $ A^{[1]^{T}} $, which is transpose of my answer.

By checking dimensions of the answer, I could verify that the answer should be $ A^{[1]^{T}} $ instead if $ A^{[1]} $. But is there any general rule for such cases using which I could directly tell whether the derivative will be the transpose of a matrix or not without verifying dimensions?

I've also checked matrix cookbook but couldn't find any related thumb rules.

$\endgroup$

1 Answer 1

8
$\begingroup$

Unfortunately, there are 2 common convention in matrix calculus: Jacobian (Numerator) and Gradient (Denominator).

For a function $f:\mathbb R^n \to \mathbb R^m$, then

  1. Jacobian convention: $\frac{\partial f}{\partial x}$ is represented by the $m\times n$ matrix with entries $\big(\tfrac{\partial f}{\partial x}\big)_{ij} = \tfrac{\partial f_i}{\partial x_j}$

  2. Gradiant convention: $\frac{\partial f}{\partial x}$ is represented by the $n\times m$ matrix with entries $\big(\tfrac{\partial f}{\partial x}\big)_{ij} = \tfrac{\partial f_j}{\partial x_i}$

If $f(x) = Ax$ then one can easily check that

  1. Under Jacobian convention $\frac{\partial f}{\partial x} = A$

  2. Under Gradient convention $\frac{\partial f}{\partial x} = A^T$

So it is a matter of which convention you are following.

$\endgroup$
4
  • $\begingroup$ Thanks for the answer! So if I don't know which convention is used, I've to verify this using only checking dimensions, right? Can you provide any references for these conventions? Thanks. $\endgroup$
    – Kaushal28
    Jul 8, 2019 at 16:07
  • $\begingroup$ @Kaushal28 en.wikipedia.org/wiki/Matrix_calculus#Layout_conventions $\endgroup$
    – Hyperplane
    Jul 8, 2019 at 17:06
  • $\begingroup$ @Kaushal28 Also, I highly recommend matrixcalculus.org which is a new CAS that can do matrix calculus; very useful for checking hand computations. $\endgroup$
    – Hyperplane
    Jul 8, 2019 at 17:09
  • $\begingroup$ Got it! A detailed explanation of matrix calculus is given here: explained.ai/matrix-calculus/index.html $\endgroup$
    – Kaushal28
    Jul 9, 2019 at 15:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .