In an example of Principle Component Analysis, my textbook uses vector calculus to do the following:

$\nabla_{\mathbf{c}}(-2\mathbf{x}^T \mathbf{D} \mathbf{c} + \mathbf{c}^T \mathbf{c}) = \mathbf{0}$

$\rightarrow -2\mathbf{D}^T \mathbf{x} + 2\mathbf{c} = \mathbf{0}$

$\rightarrow c = \mathbf{D}^T \mathbf{x}$

Where $\nabla_{\mathbf{c}}$ is the gradient with respect to $\mathbf{c}$, $\mathbf{D} \in \mathbb{R}^{n \times l}$, $\mathbf{c} \in \mathbb{R}^l$, and the columns of $\mathbf{D}$ are orthogonal to each other.

I have the following questions:

  1. How did the authors get from $\mathbf{x}^T \mathbf{D} \mathbf{c}$ to $\mathbf{D}^T \mathbf{x}$? I haven't studied matrix calculus, but I'm assuming that $\dfrac{ \partial }{\partial{\mathbf{c}}} \mathbf{c} = I$? What about $\mathbf{x}^T \mathbf{D}$ to $\mathbf{D}^T \mathbf{x}$?

  2. How did the authors go from $\mathbf{c}^T \mathbf{c}$ to $2\mathbf{c}$? I found the following in Matrix Differentiation by Randal J. Barnes:

enter image description here

If we assume that $\mathbf{A} = I$, then isn't this what we're looking for? But wouldn't that leave us with $2\mathbf{c}^T$ instead of just $2\mathbf{c}$?

I would greatly appreciate it if people could please take the time to clarify this.

  • $\begingroup$ The gradient of the inner product $\langle \mathbf a, \mathbf x \rangle$ with respect to $\bf x$ is $\bf a$. This answers your 1st question. $\endgroup$ – Rodrigo de Azevedo Jul 10 '18 at 9:20
  • $\begingroup$ @RodrigodeAzevedo Thanks for the response. I don’t understand what the inner product has to do with this? $\endgroup$ – The Pointer Jul 10 '18 at 9:28
  • 1
    $\begingroup$ $\mathbf{x}^\top \mathbf{D} \,\mathbf{c} = \langle \mathbf D^\top \mathbf x, \mathbf c \rangle$. $\endgroup$ – Rodrigo de Azevedo Jul 10 '18 at 9:31
  • $\begingroup$ @RodrigodeAzevedo So you're saying that $\dfrac{ \partial }{\partial{\mathbf{c}}} \mathbf{x}^T \mathbf{D} \mathbf{c} = \mathbf{D}^T \mathbf{x}$? $\endgroup$ – The Pointer Jul 10 '18 at 9:43
  • $\begingroup$ Yes, indeed I am. $\endgroup$ – Rodrigo de Azevedo Jul 10 '18 at 9:44

There are (unfortunately) different conventions in matrix calculus. The most common is $$\nabla_{c} \equiv \frac{\partial}{\partial c}$$ However, in you example the convention is $$\nabla_{c} \equiv \frac{\partial}{\partial c^{T}}$$ With this in mind, we perform the calculation \begin{align*} \frac{\partial}{\partial c^{T}}\left(-2x^{T}Dc + c^{T}c\right) = 0 \tag{1}\\ \frac{\partial}{\partial c^{T}}\left(-2c^{T}D^{T}x + c^{T}c\right) = 0 \tag{2}\\ -2\frac{\partial c^{T}}{\partial c^{T}}D^{T}x + \frac{\partial}{\partial c^{T}}(c^{T}c) = 0 \tag{3}\\ -2D^{T}x + \frac{\partial c^{T}}{\partial c^{T}}c + c^{T}\frac{\partial c}{\partial c^{T}}= 0 \tag{4}\\ -2D^{T}x + c + c^{T}\frac{\partial c}{\partial c^{T}}= 0 \tag{5}\\ -2D^{T}x + 2c= 0 \tag{6} \end{align*}

Now, to get from from $(5)$ to $(6)$ we need to show that $\displaystyle c^{T}\frac{\partial c}{\partial c^{T}} = c$.

We can see this using index notation. Here, $c$ has an upper index, and $c^{T}$ has a lower index. They are related by $c^{i} = \delta^{ij}(c^{T})_{j}$ and $(c^{T})_{i} = \delta_{ij}c^{j}$ (where $\delta$ is the kronecker delta).

So we have $$\left(c^{T}\frac{\partial c}{\partial c^{T}}\right)^j = (c^{T})_{i}\frac{\partial c^{i}}{\partial (c^{T})_j}\\ = (c^{T})_{i}\frac{\partial (c^{T})_{k}\delta^{ki}}{\partial (c^{T})_j}\\ = (c^{T})_{i}\frac{\partial (c^{T})_{k}}{\partial (c^{T})_j}\delta^{ki}\\ = (c^{T})_{i}\delta^{j}_{k}\delta^{ki} = (c^{T})_{i}\delta^{ji} = c^{j} $$

And we are done.

  • $\begingroup$ Thanks for the response. In the second part, you substituted $c^i = (c^{T})_{k}\delta^{ki}$, but, since you said that $c^{i} = \delta^{ij}(c^{T})_{j}$, shouldn't it be $(c^{T})_{k}\delta^{ik}$? $\endgroup$ – The Pointer Jul 10 '18 at 12:40
  • 1
    $\begingroup$ Oh, the Kronecker deltas are defined by $$\delta^{ij} = \begin{cases}1&i=j\\0&i\neq j\end{cases}$$ hence they are symmetric. If you represent the objects with upper indices by column matrices, and the objects with lower indices by lower indices, you get that indeed $(c^{T})_{i}$ and $c^{j}$ are the transposes of each other, $\delta^{ij}$ turns row matrices into column matrices, $\delta_{ij}$ turns column matrices into row matrices, and $\delta^{i}_{j}$ is the identity matrix. $\endgroup$ – Jackozee Hakkiuz Jul 10 '18 at 17:16
  • $\begingroup$ I understand now. Thank you very much for the clarification. $\endgroup$ – The Pointer Jul 10 '18 at 17:25
  • 1
    $\begingroup$ You're welcome. You can read more here about the different conventions in matrix calculus. $\endgroup$ – Jackozee Hakkiuz Jul 10 '18 at 17:29
  • 1
    $\begingroup$ I meant ...the objects with lower indices by row matrices... $\endgroup$ – Jackozee Hakkiuz Jul 10 '18 at 19:25

One can define the gradient $\nabla f$ of a (nice) scalar function $f$ as the unique vector valued function such that $$ f(x+v) = f(x) + \nabla f(x)^Tv + o(|v|) \qquad (|v|\to 0)$$ From this, one can prove the following basic results-

  1. If $f(x)$ is a linear map given by matrix multiplication $f(x) = Ax$ then its gradient is constant in $x$, with value $\nabla f(x) =A^T$.
  2. If $f(x) = x^TAx$ then $\nabla f(x) = (A^T+A)x $.

I would say these are good exercises if you haven't worked with derivatives much. These together give the result.

A remark about the result you found in Barnes's book: he is using a different convention, $$f(x+v) = f(x) + \tilde\nabla f(x) v + o(|v|)$$ and these are of course related by transposition of matrices. But you should take note which convention you are using when looking at other resources.

  • $\begingroup$ Thanks for the response. I have a few questions: [...] $\endgroup$ – The Pointer Jul 10 '18 at 9:49
  • $\begingroup$ 1. What is $o(|v|) \qquad (|v|\to 0)$? Is that supposed to be "little o" notation? I'm familiar with "big O" notation. $\endgroup$ – The Pointer Jul 10 '18 at 9:50
  • $\begingroup$ 2. Why is your result for $\nabla f(x) = (A^T+A)x$ different from that of Barnes? How did you come to this conclusion instead of his $x^T(A^T+A)$? $\endgroup$ – The Pointer Jul 10 '18 at 9:51
  • $\begingroup$ @ThePointer 1 yes, we say $f = o(g)$ as $v\to S$ if $\lim_{v\to S} f/g = 0$. You can think of it as 'going to zero faster than $g$', and for many nice functions its almost the same as $O(|v|^2)$. $\endgroup$ – Calvin Khor Jul 10 '18 at 9:52
  • $\begingroup$ @ThePointer 2. See remark at bottom of answer. $\endgroup$ – Calvin Khor Jul 10 '18 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.