0
$\begingroup$

Note: just in case any of this notation seems wrong or or something, see matrix cookbook: p.15's (141), p.9's (57) and p.8's (43)

Prove that $\forall \ p \ \epsilon \ \mathbb{N}, \forall \ X \ \epsilon \ \mathbb{R}^{pxp}, \ $ if X is a positive definite matrix, $\frac{\partial ln|X|}{\partial X} = 2X^{-1} - diag(X^{-1})$.

In words: derivative of logarithm of determinantof a matrix with respect to the matrix is twice inverse minus diagonal matrix of the inverse

What I tried:

Firstly, we define differentiation of a scalar function $u$ (I guess $u: \mathbb R \to \mathbb R$) with respect to a matrix $X$ (I guess the following definition doesn't rely on $X$ positive definite, but I might be wrong):

$\frac{\partial u}{\partial X} := \begin{bmatrix} \frac{\partial u}{\partial X_{11}} \cdots \frac{\partial u}{\partial X_{1p}}\\ \vdots \ \ddots \ \vdots \\ \frac{\partial u}{\partial X_{p1}} \cdots \frac{\partial u}{\partial X_{pp}} \end{bmatrix}$, assuming of course each entry is defined.

Thus, $\frac{\partial ln|X|}{\partial X} = \begin{bmatrix} \frac{\partial ln|X|}{\partial X_{11}} \cdots \frac{\partial ln|X|}{\partial X_{1p}}\\ \vdots \ \ddots \ \vdots \\ \frac{\partial ln|X|}{\partial X_{p1}} \cdots \frac{\partial ln|X|}{\partial X_{pp}} \end{bmatrix}$

We first note that for the case where the elements of X are independent, a constructive proof involving cofactor expansion and adjoint matrices can be made to show that $\frac{\partial ln|X|}{\partial X} = X^{-T}$ (Harville). This is not always equal to $2X^{-1}-diag(X^{-1})$. The fact alone that X is positive definite is sufficient to conclude that X is symmetric and thus its elements are not independent.

It can be shown $\frac{\partial ln|X|}{\partial X_{ij}}=tr[X^{-1} \frac{\partial X}{\partial X_{ij}}]$. I prove this here and here.

Observe that

$\frac{\partial X}{\partial X_{ij}}$ is matrix with 1 in its ith row and jth column and 0 elsewhere if i=j

and $\frac{\partial X}{\partial X_{ij}}$ is a matrix with 1 in its ith row and jth column and its jth row and ith column (since positive definite matrices are symmetric) otherwise.

Examples:

$\frac{\partial X}{\partial X_{11}} = \begin{bmatrix} \frac{\partial X_{11}}{\partial X_{11}} \cdots \frac{\partial X_{1p}}{\partial X_{11}}\\ \vdots \ \ddots \ \vdots \\ \frac{\partial X_{p1}}{\partial X_{11}} \cdots \frac{\partial X_{pp}}{\partial X_{11}} \end{bmatrix}$

$= \begin{bmatrix} 1 \ 0 \cdots 0\\ 0 \ 0 \cdots 0\\ \vdots \ \vdots \ \ddots \ \vdots \\ 0 \ 0 \cdots 0 \end{bmatrix}$

$\frac{\partial X}{\partial X_{12}} = \begin{bmatrix} \frac{\partial X_{11}}{\partial X_{12}} \cdots \frac{\partial X_{1p}}{\partial X_{12}}\\ \vdots \ \ddots \ \vdots \\ \frac{\partial X_{p1}}{\partial X_{12}} \cdots \frac{\partial X_{pp}}{\partial X_{12}} \end{bmatrix}$

$= \begin{bmatrix} 0 \ 1 \ 0 \cdots 0\\ 1 \ 0 \ 0 \cdots 0\\ 0 \ 0 \ 0 \cdots 0\\ \vdots \ \vdots \ \vdots \ \ddots \ \vdots \\ 0 \ 0 \ 0 \cdots 0 \end{bmatrix}$

since $\frac{\partial X_{12}}{\partial X_{12}} = \frac{\partial X_{12}}{\partial X_{21}}$ since $X_{21}=X_{12}$.

Thus, if we let $E = X^{-1}= \begin{bmatrix}e_{11} \cdots e_{1p}\\ \vdots \ \ddots \ \vdots \\ e_{p1} \cdots e_{pp} \end{bmatrix}$

and if we let $F_{ij} = X^{-1} \frac{\partial X}{\partial X_{ij}} = E \frac{\partial X}{\partial X_{ij}}$,then $F_{ij}$ is a matrix containing $e_{ij}$ in the ith row and jth column and zero elsewhere if i=j and is a matrix containing $e_{ij}$, $e_{ji}$ and zeroes in the main diagonal and zero elsewhere otherwise.

If we let $F = [f_{ij}] = tr(F_{ij})$, then $f_{ij} = \frac{\partial ln|X|}{\partial X_{ij}} = e_{ij}$ if i=j and $\frac{\partial ln|X|}{\partial X_{ij}} = tr(F_{ij}) = 2e_{ij}$ otherwise. (Note: $F \ne [F_{ij}]$. Probably should've used better notation.)

Example:

$F_{12} = E \frac{\partial X}{\partial X_{12}} = \begin{bmatrix} e_{11} \cdots e_{1p}\\ \vdots \ \ddots \ \vdots \\ e_{p1} \cdots e_{pp} \end{bmatrix}\begin{bmatrix} \frac{\partial X_{11}}{\partial X_{12}} \cdots \frac{\partial X_{1p}}{\partial X_{12}}\\ \vdots \ \ddots \ \vdots \\ \frac{\partial X_{p1}}{\partial X_{12}} \cdots \frac{\partial X_{pp}}{\partial X_{12}} \end{bmatrix}$

$=\frac{\partial X}{\partial X_{12}} = \begin{bmatrix} e_{11} \cdots e_{1p}\\ \vdots \ \ddots \ \vdots \\ e_{p1} \cdots e_{pp} \end{bmatrix}\begin{bmatrix} 0 \ 1 \ 0 \cdots 0\\ 1 \ 0 \ 0 \cdots 0\\ 0 \ 0 \ 0 \cdots 0\\ \vdots \ \vdots \ \vdots \ \ddots \ \vdots \\ 0 \ 0 \ 0 \cdots 0 \end{bmatrix}$

$=\begin{bmatrix} e_{12} \ 0 \ 0 \cdots 0\\ 0 \ e_{21} \ 0 \cdots 0\\ 0 \ 0 \ 0 \cdots 0\\ \vdots \ \vdots \ \vdots \ \ddots \ \vdots \\ 0 \ 0 \ 0 \cdots 0 \end{bmatrix}$

Thus, $tr(F_{12})=e_{12}+e_{21} = 2e_{12} = 2e_{21}$

$\therefore, \frac{\partial ln|X|}{\partial X} = F = \begin{bmatrix} e_{11} \ 2e_{12} \cdots 2e_{1p}\\ 2e_{21} \ e_{22} \cdots 2e_{2p}\\ \vdots \ \vdots \ \ddots \ \vdots \\ 2e_{p1} \ 2e_{p2} \cdots e_{pp} \end{bmatrix}= 2E -$ diag(E) = $2X^{-1} -$ diag($X^{-1}$).

QED

Any mistakes? Are there simpler or alternative ways?

$\endgroup$
1
  • $\begingroup$ Perhaps you could elaborate what you mean by $\frac{\partial ln|X|}{\partial X}$, in particular the underlying inner product. $\endgroup$ – copper.hat Apr 16 at 19:31
1
$\begingroup$

Perhaps I am missing something, but the result does not seem correct just from a specific example.

Let $f(X) = \log(\det(X))$.

I am assuming when you write $\frac{\partial ln|X|}{\partial X}$ that you mean that the derivative of $f$ evaluated at $X$ in the direction $H$ is given by $Df(X)H = \operatorname{tr}(\frac{\partial ln|X|}{\partial X}^{-1} H)$.

If this is not correct, perhaps you could add your definition to avoid causing confusion and I will delete this answer.

Choose $X=\begin{bmatrix} 1 & 1 \\ 1 & 2 \end{bmatrix}$ and $H=\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$. Both are positive definite and symmetric and $X^{-1} = \begin{bmatrix} 2 & -1 \\ -1 & 1 \end{bmatrix}$.

The usual formula gives $Df(X)H = \lim_{t\to 0} {f(X+tH)-f(X) \over t} = \operatorname{tr}(X^{-1} H)$.

With the above matrices we get $Df(X)H =4$ and by taking $t=0.01$ and just to confirm, evaluating numerically using the quotient we get ${f(X+tH)-f(X) \over t} \approx 3.95$ which is close to $4$.

From the above formula we have $G= 2X^{-1}-\operatorname{diag} X^{-1} = \begin{bmatrix} 2 & -2 \\ -2 & 1 \end{bmatrix}$ and $\operatorname{tr}(G H) = 2$.

$\endgroup$
17
  • $\begingroup$ man it's been 5.5 years. i don't remember, hehe. which part exactly is wrong please? re the trace thing i prove in the other links math.stackexchange.com/questions/1493141/… or math.stackexchange.com/questions/1493139/… $\endgroup$ – BCLC Apr 17 at 1:02
  • $\begingroup$ The formula in the question is incorrect if one is using the trace inner product. $\endgroup$ – copper.hat Apr 17 at 1:16
  • $\begingroup$ perhaps the proofs in the other questions can tell you what the statement means? like under interpretation 1, the proof is wrong but under interpretation 2 the proof is right, then probably it's interpretation 2? $\endgroup$ – BCLC Apr 17 at 5:09
  • 1
    $\begingroup$ @BCLC I disagree with notation. The notation ${\partial \over \partial x}$ is typically used for the Fréchet derivative. The formula is the cookbook is for the derivative with respect to some structure, see (134). Personally I think this is very misleading. $\endgroup$ – copper.hat Apr 17 at 6:58
  • 1
    $\begingroup$ anyway i'm just gonna accept your answer. $\endgroup$ – BCLC Apr 17 at 10:58
1
$\begingroup$

If we just use the chain rule, positivity of $det$ for PD matrices, and the formula for the derivative of $det$ for invertible matrices, then $[d(\ln\det X)](H)=\operatorname{tr}(X^{-1}H)$ as per Compute the derivative of the log of the determinant of A with respect to A, where one recognizes the Frobenius inner product with the gradient $X^{-T}$ in agreement with @loup blanc answer above and with "Harville".

$\endgroup$
2
  • $\begingroup$ Ummmmmmmm what? $\endgroup$ – BCLC Oct 23 '15 at 11:35
  • 1
    $\begingroup$ @ rych , $|X|$ denotes $\det(X)$. Moreover your result should be $X/||X||^2$. $\endgroup$ – user91684 Oct 23 '15 at 17:04
1
$\begingroup$

@ BCLC , you notation $\frac{\partial ln|X|}{\partial X}$ is a bad one because the entries of $X$ are not independent. Where did you find this formula ?

Let $f:X\in GL_n\rightarrow \log(|\det(X)|), g:X\rightarrow \det(X)$. Since, for every $H\in M_n$, $Dg_X(H)=\det(X)tr(X^{-1}H)$, $Df_X(H)=tr(X^{-1}H)$. Using the standard inner product over $M_n$, the gradient of $f$ is defined by $Df_X(H)=<\nabla f(X),H>=tr((\nabla f(X))^TH)$; then $\nabla f(X)=X^{-T}$.

Now, let $\phi:Y\in S_n^+\rightarrow \log(|\det(Y)|)$, where $S_n^+$ is the set of SPD matrices. Since the tangent vector space to $S_n^+$ is $S_n$, for every $K\in S_n$, $D\phi_Y(K)=tr(Y^{-1}K)$. The restriction to $S_n$ of the previous inner product is an inner product. Then, for this inner product and $K\in S_n$, $\nabla \phi(Y)=Y^{-1}$.

Of course, we find the same formula because the Taylor series of $f,\phi$ are $f(X+H)=f(X)+tr(X^{-1}H)+o(||H||)$ and $\phi(Y+K)=\phi(Y)+tr(Y^{-1}K)+o(||K||)$, that is not extraordinary because the restriction of $f$ is $\phi$.

EDIT. Answer to BCLC. There are standard definitions concerning the derivative and the gradient of a real function. Here we consider a curious third definition:

Let $f:GL_n^+\rightarrow \mathbb{R}$, $\frac{\partial f}{\partial X}=G$ and $\phi$ its restriction to $S_n^+$, $\frac{\partial \phi}{\partial X}=\Gamma$. Here, $\Gamma$ is constructed as follows: if $i\not= j$, then $\gamma_{i,j}=\frac{\partial f}{\partial X_{i,j}}+\frac{\partial f}{\partial X_{j,i}}=2g_{i,j}$ and otherwise $\gamma_{i,i}=\frac{\partial f}{\partial X_{i,i}}=g_{i,i}$. In a standard way, we derive and, in a second step, we put $X_{i,j}=X_{j,i}$. Here we do the contrary: we put $X_{i,j}=X_{j,i}$ and, in a second step, we derive.

Anyway, we obtain the following interesting result: the gradient of the restriction is not the restriction of the gradient.

$\endgroup$
8
  • $\begingroup$ Th 2.14d in Rencher and Schaalje Also ummmmmmmm what? $\endgroup$ – BCLC Oct 23 '15 at 22:14
  • $\begingroup$ @ BCLC , I hope you can write something else that "um what ?". The formula is also in the matrixcookbook 2.8.2 formula 130, that does not surprise me. Since a gradient is associated to an inner product, for me, this formula is non-sense. How do you correctly use the Taylor formula ? Anyway, you can possibly derive with respect to the upper part of the symmetric matrix. Now, you're a big boy; do as you want. $\endgroup$ – user91684 Oct 23 '15 at 22:59
  • $\begingroup$ I have no idea what your answer is supposed to say. None of those things are used in the Rencher and Schaalje book up to the Th 2.14d, afaik. $\endgroup$ – BCLC Oct 23 '15 at 23:57
  • 1
    $\begingroup$ Yes, the author of the formula did the same calculation as you. Now, the question is: what are the geometric significance and the interest of such a result ? $\endgroup$ – user91684 Oct 26 '15 at 18:51
  • 1
    $\begingroup$ Yes, of course. $\endgroup$ – user91684 Oct 26 '15 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.