1
$\begingroup$

I am trying to understand a derivation and there is a matrix manipulation which I do not understand. So, there is the following derivative:

$$ \frac{d}{dx} (x^T\Sigma^{-1}x) $$

Here x is a D $\times$ 1 vector (a D-dimensional vector) and $\Sigma$ is an invertible diagonal matrix. So, the derivation proceeds to say that the above quantity is:

$$ 2 \Sigma^{-1}x $$

If x was a scalar, I can see how this follows by chain rule but I got confused as to how it applies with the matrix transpose going on there. I was wondering if someone can point me to what identities are being used.

$\endgroup$
  • $\begingroup$ What definition are you using for the derivative of a scalar quantity with respect to a vector? $\endgroup$ – Semiclassical Sep 23 '14 at 14:56
  • $\begingroup$ a more correct answer is $2x^T\Sigma^{-1}$, a row vector $\endgroup$ – rych Oct 21 '14 at 4:40
1
$\begingroup$

Think of it coordinate-by-coordinate, then.

$$f(x_1,\ldots, x_n) = x^T \Sigma^{-1} x = \sum_{i,j} x_i (\Sigma^{-1})_{ij} x_j$$

For a particular component, say $x_k$, you have terms where $i=k$ and terms where $j=k$.

$$ \dfrac{\partial f}{\partial x_k} = \sum_j (\Sigma^{-1})_{kj} x_j + \sum_i x_i (\Sigma^{-1})_{ik} = \sum_j \left((\Sigma^{-1})_{kj} + (\Sigma^{-1})_{jk})\right) x_j $$ and now use the fact that $\Sigma^{-1}$ is symmetric $$ = 2 \sum_{j} (\Sigma^{-1})_{kj} x_j = 2 \left( \Sigma^{-1} x \right)_k $$

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

A mostly used identity $\frac{1}{\begin{pmatrix}v_1\\v_2\\..\\v_n\end{pmatrix}}=\frac{1}{n} \begin{pmatrix}v_1^{-1}\\v_2^{-1}\\..\\v_n^{-1}\end{pmatrix}$

With this $\left(\frac{1}{\vec v}\right)^\top\cdot \vec v = 1$

$$\frac{d}{d\vec x} \left(\vec x^\top\Sigma^{-1}\vec x\right) =\frac{1}{n} (\partial_{x_1}, \partial_{x_2}, \partial_{x_3}, ..., \partial_{x_n})^\top \left(\vec x^\top\Sigma^{-1}\vec x\right)$$ $$ \Sigma^{-1}\vec x +\frac{d}{dx}\vec x^\top\Sigma^{-1}\vec x$$

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Write the term $x^T\Sigma^{−1}x$ explicity as $\sum_i\sum_i x_ix_j\sigma_{ii}^*$ here $\Sigma^{-1}=(\sigma_{ij}^*)$. Then you have to calculate the gradient of this expression with respect to the vector $x$. Then you see it.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Let $A=\Sigma^{-1}$. In order to find the derivative let us calculate the h-linear term, $$\begin{equation} \begin{split} (x+h)^TA(x+h)-x^TAx&=(x^T+h^T)A(x+h)-x^TAx\approx\\ &h^TAx+x^TAh= (h^TAx)^T+x^TAh=\\&x^T(A+A^T)h \end{split} \end{equation}$$ Hence the derivative is: $x^T(A+A^T)$, or $2x^TA$ for a symmetric $A$. Note that it is not a column- but a row-vector (or covector, or dual vector).

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.