# Confusion with Matrix calculus derivative computation.

The text I am reading has the following

$$E=(I+\nabla I\frac{\partial W}{\partial p}\Delta p -T)^2$$ where $\nabla I$ us a row vector $1$ by $2$, $\frac{\partial W}{\partial p}$ is a $2$ by $3$ matrix and $\Delta p$ is a $3$ by $1$ column vector. $I$ and $T$ are scalars. We seek the gradient $\frac{\partial E}{\partial \Delta p_k}$, but I do not understand their solution. I will present my attempt, and than their solution.

In index notation, we have $(I+\nabla I_i\frac{\partial W^i}{\partial \Delta p_k}\Delta p^k -T)^2$, where summation if implied by repeated indicies, Than the derivative is

$$\frac{\partial E}{\partial \Delta p_m}=2(I+\nabla I_i\frac{\partial W^i}{\partial p_k}\Delta p^k -T)\nabla I_n\frac{\partial W^n}{\partial p_m}$$ or returning to the matrix notation

$$\frac{\partial E}{\partial \Delta p}=2(I+\nabla I\frac{\partial W}{\partial p}\Delta p -T)\nabla I\frac{\partial W}{\partial p}\tag{1}\label{1}$$

But their solution is

$$\frac{\partial E}{\partial \Delta p}=2(\nabla I\frac{\partial W}{\partial p})^T(I+\nabla I\frac{\partial W}{\partial p}\Delta p -T)\tag{2}\label{2}$$

Why is their solution correct and where did I make the mistake? My solution is equation \eqref{1}, but theirs is \eqref{2}. I am confused, why in theirs, $$(\nabla I\frac{\partial W}{\partial p})$$ is transposed?

• In the last two equations you have variable $p_m$ with subindex $m$ on the left hand side, but no subscripts on the right hand side. Is there a chance there is a typo in your problem statement?
Mar 21 '18 at 2:30
• I was careless when copy and pasting. My solution in matrix form is that in equation (1), what is before, is my derivation. Their solution is equation (2). Hope this clears things up Mar 21 '18 at 2:33
• I think now you very first equation (the one without label) has non-matching subindex $p_m$ in the last term.
Mar 21 '18 at 2:35
• You are correct, I have fixed it. Thank you for sticking with me, it has been a long day :) Mar 21 '18 at 2:36
• I edited and added that information. Mar 21 '18 at 2:52

It looks like the text you are reading uses so-called Denominator layout, for matrix calculus notation, i.e. given two column vectors $\boldsymbol{x}\in\mathbb{R}^m$ and $\boldsymbol{y}\in\mathbb{R}^n$ of the size $m\times1$ and $n\times1$ respectively we write derivative $\displaystyle\dfrac{\partial \boldsymbol{y}}{\partial\boldsymbol{x}}$ as $n\times m$ matrix. In other words, the layout is according to $\boldsymbol y^{\boldsymbol\top}$ and $\mathbf{x}$.

In your case $E$ is the scalar, so that $n=1$ and $m=3$, and thus the derivative of scalar w.r.t. the vector $\Delta p$ has to be a column vector of the size $3\times1$.

More details are available, for example, on the Wikipedia page for Matrix Calculus.

Moreover, using provided there table of scalar-by-vector identities one can figure out dimensionality of each term emerging from the chain rule explicitly. According to the linked table,

Assume $\boldsymbol{b}\in \mathbb R^m$ and $\boldsymbol x \in \mathbb{R}^n$ are column vectors, and matrix $\boldsymbol A$ is in $\mathbb{R}^{m\times n}$. If $\boldsymbol A$ and $\boldsymbol b$ do not depend on $\boldsymbol x$, then the following holds (in Denominator layout ): $$\dfrac{\partial\left(\boldsymbol b^{\boldsymbol\top}\boldsymbol A \boldsymbol x\right)}{\partial \boldsymbol x} = \boldsymbol A^{\boldsymbol \top} \boldsymbol b = \left(\boldsymbol b^{\boldsymbol\top}\boldsymbol A\right)^{\boldsymbol\top}$$

Using this identity, you can easily see that $$\frac{\partial }{\partial \Delta p} \left( \nabla I\frac{\partial W}{\partial p}\Delta p\right) = \left( \nabla I\frac{\partial W}{\partial p}\right)^{\boldsymbol\top}$$

• Thank you for the reply. What I do not undestand, is where I make a mistake in my derivation? Why the term transposed in their solution. Is it because they want the gradient to be a column vector and not a row vector? I do not understand why it differes from my solution... I can get their solution from mine, by transposing mine, and than the first term in mine can remain the same, as it is a scalar, and the second transposes. Mar 21 '18 at 3:38
• @LeastSquaresWonderer As far as I understand this is just a matter of convention. If you read the wiki article I referenced, you can find there information about notation conventions, their ambiguity, and motivation behind choosing one over another.