# Matrix Calculus: $\nabla_{\mathbf{c}}(-2\mathbf{x}^T \mathbf{D} \mathbf{c} + \mathbf{c}^T \mathbf{c}) = -2\mathbf{D}^T \mathbf{x} + 2\mathbf{c}$?

In an example of Principle Component Analysis, my textbook uses vector calculus to do the following:

$\nabla_{\mathbf{c}}(-2\mathbf{x}^T \mathbf{D} \mathbf{c} + \mathbf{c}^T \mathbf{c}) = \mathbf{0}$

$\rightarrow -2\mathbf{D}^T \mathbf{x} + 2\mathbf{c} = \mathbf{0}$

$\rightarrow c = \mathbf{D}^T \mathbf{x}$

Where $\nabla_{\mathbf{c}}$ is the gradient with respect to $\mathbf{c}$, $\mathbf{D} \in \mathbb{R}^{n \times l}$, $\mathbf{c} \in \mathbb{R}^l$, and the columns of $\mathbf{D}$ are orthogonal to each other.

I have the following questions:

1. How did the authors get from $\mathbf{x}^T \mathbf{D} \mathbf{c}$ to $\mathbf{D}^T \mathbf{x}$? I haven't studied matrix calculus, but I'm assuming that $\dfrac{ \partial }{\partial{\mathbf{c}}} \mathbf{c} = I$? What about $\mathbf{x}^T \mathbf{D}$ to $\mathbf{D}^T \mathbf{x}$?

2. How did the authors go from $\mathbf{c}^T \mathbf{c}$ to $2\mathbf{c}$? I found the following in Matrix Differentiation by Randal J. Barnes:

If we assume that $\mathbf{A} = I$, then isn't this what we're looking for? But wouldn't that leave us with $2\mathbf{c}^T$ instead of just $2\mathbf{c}$?

I would greatly appreciate it if people could please take the time to clarify this.

• The gradient of the inner product $\langle \mathbf a, \mathbf x \rangle$ with respect to $\bf x$ is $\bf a$. This answers your 1st question. – Rodrigo de Azevedo Jul 10 '18 at 9:20
• @RodrigodeAzevedo Thanks for the response. I don’t understand what the inner product has to do with this? – The Pointer Jul 10 '18 at 9:28
• $\mathbf{x}^\top \mathbf{D} \,\mathbf{c} = \langle \mathbf D^\top \mathbf x, \mathbf c \rangle$. – Rodrigo de Azevedo Jul 10 '18 at 9:31
• @RodrigodeAzevedo So you're saying that $\dfrac{ \partial }{\partial{\mathbf{c}}} \mathbf{x}^T \mathbf{D} \mathbf{c} = \mathbf{D}^T \mathbf{x}$? – The Pointer Jul 10 '18 at 9:43
• Yes, indeed I am. – Rodrigo de Azevedo Jul 10 '18 at 9:44

There are (unfortunately) different conventions in matrix calculus. The most common is $$\nabla_{c} \equiv \frac{\partial}{\partial c}$$ However, in you example the convention is $$\nabla_{c} \equiv \frac{\partial}{\partial c^{T}}$$ With this in mind, we perform the calculation \begin{align*} \frac{\partial}{\partial c^{T}}\left(-2x^{T}Dc + c^{T}c\right) = 0 \tag{1}\\ \frac{\partial}{\partial c^{T}}\left(-2c^{T}D^{T}x + c^{T}c\right) = 0 \tag{2}\\ -2\frac{\partial c^{T}}{\partial c^{T}}D^{T}x + \frac{\partial}{\partial c^{T}}(c^{T}c) = 0 \tag{3}\\ -2D^{T}x + \frac{\partial c^{T}}{\partial c^{T}}c + c^{T}\frac{\partial c}{\partial c^{T}}= 0 \tag{4}\\ -2D^{T}x + c + c^{T}\frac{\partial c}{\partial c^{T}}= 0 \tag{5}\\ -2D^{T}x + 2c= 0 \tag{6} \end{align*}

Now, to get from from $(5)$ to $(6)$ we need to show that $\displaystyle c^{T}\frac{\partial c}{\partial c^{T}} = c$.

We can see this using index notation. Here, $c$ has an upper index, and $c^{T}$ has a lower index. They are related by $c^{i} = \delta^{ij}(c^{T})_{j}$ and $(c^{T})_{i} = \delta_{ij}c^{j}$ (where $\delta$ is the kronecker delta).

So we have $$\left(c^{T}\frac{\partial c}{\partial c^{T}}\right)^j = (c^{T})_{i}\frac{\partial c^{i}}{\partial (c^{T})_j}\\ = (c^{T})_{i}\frac{\partial (c^{T})_{k}\delta^{ki}}{\partial (c^{T})_j}\\ = (c^{T})_{i}\frac{\partial (c^{T})_{k}}{\partial (c^{T})_j}\delta^{ki}\\ = (c^{T})_{i}\delta^{j}_{k}\delta^{ki} = (c^{T})_{i}\delta^{ji} = c^{j}$$

And we are done.

• Thanks for the response. In the second part, you substituted $c^i = (c^{T})_{k}\delta^{ki}$, but, since you said that $c^{i} = \delta^{ij}(c^{T})_{j}$, shouldn't it be $(c^{T})_{k}\delta^{ik}$? – The Pointer Jul 10 '18 at 12:40
• Oh, the Kronecker deltas are defined by $$\delta^{ij} = \begin{cases}1&i=j\\0&i\neq j\end{cases}$$ hence they are symmetric. If you represent the objects with upper indices by column matrices, and the objects with lower indices by lower indices, you get that indeed $(c^{T})_{i}$ and $c^{j}$ are the transposes of each other, $\delta^{ij}$ turns row matrices into column matrices, $\delta_{ij}$ turns column matrices into row matrices, and $\delta^{i}_{j}$ is the identity matrix. – Jackozee Hakkiuz Jul 10 '18 at 17:16
• I understand now. Thank you very much for the clarification. – The Pointer Jul 10 '18 at 17:25
• You're welcome. You can read more here about the different conventions in matrix calculus. – Jackozee Hakkiuz Jul 10 '18 at 17:29
• I meant ...the objects with lower indices by row matrices... – Jackozee Hakkiuz Jul 10 '18 at 19:25

One can define the gradient $\nabla f$ of a (nice) scalar function $f$ as the unique vector valued function such that $$f(x+v) = f(x) + \nabla f(x)^Tv + o(|v|) \qquad (|v|\to 0)$$ From this, one can prove the following basic results-

1. If $f(x)$ is a linear map given by matrix multiplication $f(x) = Ax$ then its gradient is constant in $x$, with value $\nabla f(x) =A^T$.
2. If $f(x) = x^TAx$ then $\nabla f(x) = (A^T+A)x$.

I would say these are good exercises if you haven't worked with derivatives much. These together give the result.

A remark about the result you found in Barnes's book: he is using a different convention, $$f(x+v) = f(x) + \tilde\nabla f(x) v + o(|v|)$$ and these are of course related by transposition of matrices. But you should take note which convention you are using when looking at other resources.

• Thanks for the response. I have a few questions: [...] – The Pointer Jul 10 '18 at 9:49
• 1. What is $o(|v|) \qquad (|v|\to 0)$? Is that supposed to be "little o" notation? I'm familiar with "big O" notation. – The Pointer Jul 10 '18 at 9:50
• 2. Why is your result for $\nabla f(x) = (A^T+A)x$ different from that of Barnes? How did you come to this conclusion instead of his $x^T(A^T+A)$? – The Pointer Jul 10 '18 at 9:51
• @ThePointer 1 yes, we say $f = o(g)$ as $v\to S$ if $\lim_{v\to S} f/g = 0$. You can think of it as 'going to zero faster than $g$', and for many nice functions its almost the same as $O(|v|^2)$. – Calvin Khor Jul 10 '18 at 9:52
• @ThePointer 2. See remark at bottom of answer. – Calvin Khor Jul 10 '18 at 9:52