Derivative with respect to a matrix

How do we start with the matrix differentiation of this kind of equation? $$V = \big[ y_t - Cx_t \big]^T R^{-1} \big[y_t - Cx_t \big]$$

here $x_t$ and $y_t$ are vectors and $C$ and $R$ are matrices. $R$ is a covariance matrix (symmetric).

How do we find the derivative of $V$ with respect of matrix $C$?

-
You could try expanding $V(C+\Delta)$ and looking for the terms that are linear in $\Delta$. Keep in mind that $DV(C)$ is a linear map from matrices to a scalar. – copper.hat Feb 26 '13 at 18:05

Let $V(C) = (Cx-y)^T R^{-1} (Cx-y)$. Then \begin{eqnarray} V(C+\Delta) &=& (Cx+\Delta x-y)^T R^{-1} (Cx+\Delta x-y) \\ &=& (Cx-y)^T R^{-1} (Cx-y) + (\Delta x)^T R^{-1} (Cx-y) +(Cx-y)^T R^{-1} \Delta x +(\Delta x)^T R^{-1} \Delta x \\ &=& V(C) + 2 (Cx-y)^T R^{-1} \Delta x +(\Delta x)^T R^{-1} \Delta x \end{eqnarray} From which we have $DV(C)(\Delta) = 2 (Cx-y)^T R^{-1} \Delta x$.

You can express the derivative as a trace, as in $DV(C)(\Delta) = 2\operatorname{tr}(x (Cx-y)^T R^{-1} \Delta)$.

-
I have a question about the $\Delta$. Shouldn't it disappear from the derivative? I think we have to do something like limit $\Delta$ tends to zero, $(V(C+\Delta) - V(C) / \Delta$, right? – Rex Roy Feb 27 '13 at 0:51
And can you also suggest some pointers from where I can practice these kinds of derivatives? – Rex Roy Feb 27 '13 at 1:28
Well, this is a point that confuses a lot of folks. The derivative (in this case) is a linear map from the original space (of matrices) to scalars. There is no easy way to represent such a mapping without a '$\Delta$' of some sort. With scalars (or mappings of the form $\mathbb{R}^n \to \mathbb{R}$) you can represent the derivative as a scalar (or an element of $\mathbb{R}^n$ respectively), however, no common convenient notation exists to represent a linear map of the form $\mathbb{R}^{n \times n} \to \mathbb{R}$. – copper.hat Feb 27 '13 at 1:35
You could 'stretch' the matrix out into a long vector and compute the corresponding representative of the derivative, but it is no more convenient that the above representation in my opinion. I don't have any references or pointers, sorry. I usually expand as above and look for linear terms. – copper.hat Feb 27 '13 at 1:36
I'm still confused about the $\Delta$. But, I have to find the value of C where the derivative is zero. How do you proceed from your equation into finding the value of C setting it to zero? – Rex Roy Feb 27 '13 at 2:09

You can calculate the differential

$$dV = (-dCx_t)^TR^{-1}(y_t-Cx_t) + (y_t-Cx_t)^TR^{-1}(-dCx_t) = -2x_t^T(dC)^TR^{-1}(y_t-Cx_t),$$ where the second step follows from $R$ being symmetric. For any tangent matrix $dC$ this gives you the change in $V$. You can plug in the delta matrix $dC_{i,j} = \delta_m(i)\delta_n(j)$ if you need the individual component derivatives $\frac{\partial V}{\partial C_{m,n}}.$

-
What rule did you use to get the first expression? Is it kind of the product rule? – Rex Roy Feb 27 '13 at 2:11