2
$\begingroup$

I'm stuck with the following problem: I have to compute the second derivative (hessian matrix) of the mahalanobis distance $$ [x-\mu]^{T} \Sigma^{-1} [x-\mu] $$ wrt to the Cholesky decomposition of the covariance matrix $$ \Lambda\Lambda^{T}=\Sigma $$ On this paper I've found the solution for the first derivative

$$ {\delta J\over\delta\Lambda} = -2 {\Sigma^{-1}} [x-\mu][x-\mu]^{T}\Sigma^{-1}\Lambda $$

Taking the lower triangle of this matrix gives the vector of the first derivatives (gradient) for the k=n*(n+1)/2 components. I haven't been able to find the k*k matrix of second derivatives.

Any help would be appreciated! Thanks!

$\endgroup$
1
$\begingroup$

Let $u=x-\mu$, $f:\Lambda\rightarrow\Lambda\Lambda^T$, $g:\Sigma\rightarrow u^T\Sigma^{-1}u$ and $h=g\circ f$.

  1. Then $Dg_{\Sigma}: H\in S_k\rightarrow -u^T\Sigma^{-1}H\Sigma^{-1}u$ and $Df_{\Lambda}:K\in T_k\rightarrow K\Lambda^T+\Lambda K^T$, where $S_k$ is the set of symmetric matrices and $T_k$ is the set of lower triangular matrices. Thus $Dh_{\Lambda}:K\rightarrow -u^T\Sigma^{-1}(K\Lambda^T+\Lambda K^T)\Sigma^{-1}u=-2tr(\Lambda^T\Sigma^{-1}uu^T\Sigma^{-1}K)=-2<\Sigma^{-1}uu^T\Sigma^{-1}\Lambda,K>$

    where $<Y,Z>$ is the real scalar product $tr(Y^TZ)$.

Let $\nabla (h)_{\Lambda}=-2\Sigma^{-1}uu^T\Sigma^{-1}\Lambda$; we'll say that the previous function is the gradient of $h$ because, for every $i\geq j$, $\dfrac{\partial h}{\partial \Lambda_{ij}}=\nabla(h)_{i,j}$; the strictly upper part of $\nabla (h)$ is useless.

  1. The second derivative is the following symmetric bilinear form:

$D^2h_{\Lambda}:(K,L)\in T_k\times T_k\rightarrow 2u^T\Sigma^{-1}(L\Lambda^T+\Lambda L^T)\Sigma^{-1}(K\Lambda^T+\Lambda K^T)\Sigma^{-1}u-u^T\Sigma^{-1}(KL^T+LK^T)\Sigma^{-1}u$.

For example, if $p\geq q,r\geq s$, then $\dfrac{\partial^2 h}{\partial \Lambda_{pq}\partial\Lambda_{rs}}=D^2h_{\Lambda}(E_{pq},E_{rs})$.

Note that the Hessian of $h$ is a complicated tensor .

$\endgroup$
1
  • $\begingroup$ Thanks for the answer, it is of great help. I came with a similar solution (i.e. using the chain rule), even if I did not formulate it so nicely. For more information on this topic, people that are interested can check this paper from Thomas Minka research.microsoft.com/en-us/um/people/minka/papers/matrix/… $\endgroup$ – Vincent Moens Jul 11 '16 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.