Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

How do we start with the matrix differentiation of this kind of equation? $$ V = \big[ y_t - Cx_t \big]^T R^{-1} \big[y_t - Cx_t \big] $$

here $x_t$ and $y_t$ are vectors and $C$ and $R$ are matrices. $R$ is a covariance matrix (symmetric).

How do we find the derivative of $V$ with respect of matrix $C$?

share|cite|improve this question
You could try expanding $V(C+\Delta)$ and looking for the terms that are linear in $\Delta$. Keep in mind that $DV(C)$ is a linear map from matrices to a scalar. – copper.hat Feb 26 '13 at 18:05

Let $V(C) = (Cx-y)^T R^{-1} (Cx-y)$. Then \begin{eqnarray} V(C+\Delta) &=& (Cx+\Delta x-y)^T R^{-1} (Cx+\Delta x-y) \\ &=& (Cx-y)^T R^{-1} (Cx-y) + (\Delta x)^T R^{-1} (Cx-y) +(Cx-y)^T R^{-1} \Delta x +(\Delta x)^T R^{-1} \Delta x \\ &=& V(C) + 2 (Cx-y)^T R^{-1} \Delta x +(\Delta x)^T R^{-1} \Delta x \end{eqnarray} From which we have $DV(C)(\Delta) = 2 (Cx-y)^T R^{-1} \Delta x$.

You can express the derivative as a trace, as in $DV(C)(\Delta) = 2\operatorname{tr}(x (Cx-y)^T R^{-1} \Delta)$.

share|cite|improve this answer
I have a question about the $\Delta$. Shouldn't it disappear from the derivative? I think we have to do something like limit $\Delta$ tends to zero, $(V(C+\Delta) - V(C) / \Delta$, right? – Rex Roy Feb 27 '13 at 0:51
And can you also suggest some pointers from where I can practice these kinds of derivatives? – Rex Roy Feb 27 '13 at 1:28
Well, this is a point that confuses a lot of folks. The derivative (in this case) is a linear map from the original space (of matrices) to scalars. There is no easy way to represent such a mapping without a '$\Delta$' of some sort. With scalars (or mappings of the form $\mathbb{R}^n \to \mathbb{R}$) you can represent the derivative as a scalar (or an element of $\mathbb{R}^n$ respectively), however, no common convenient notation exists to represent a linear map of the form $\mathbb{R}^{n \times n} \to \mathbb{R}$. – copper.hat Feb 27 '13 at 1:35
You could 'stretch' the matrix out into a long vector and compute the corresponding representative of the derivative, but it is no more convenient that the above representation in my opinion. I don't have any references or pointers, sorry. I usually expand as above and look for linear terms. – copper.hat Feb 27 '13 at 1:36
I'm still confused about the $\Delta$. But, I have to find the value of C where the derivative is zero. How do you proceed from your equation into finding the value of C setting it to zero? – Rex Roy Feb 27 '13 at 2:09

You can calculate the differential

$$dV = (-dCx_t)^TR^{-1}(y_t-Cx_t) + (y_t-Cx_t)^TR^{-1}(-dCx_t) = -2x_t^T(dC)^TR^{-1}(y_t-Cx_t),$$ where the second step follows from $R$ being symmetric. For any tangent matrix $dC$ this gives you the change in $V$. You can plug in the delta matrix $dC_{i,j} = \delta_m(i)\delta_n(j)$ if you need the individual component derivatives $\frac{\partial V}{\partial C_{m,n}}.$

share|cite|improve this answer
What rule did you use to get the first expression? Is it kind of the product rule? – Rex Roy Feb 27 '13 at 2:11

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.