3
$\begingroup$

I would like to differentiate the mahalanobis distance:

$$D(\textbf{x}, \boldsymbol \mu, \Sigma) = (\textbf{x}-\boldsymbol \mu)^T\Sigma^{-1}(\textbf{x}-\boldsymbol \mu)$$

where $\textbf{x} = (x_1, ..., x_n) \in \mathbb R^n, \;\boldsymbol \mu = (\mu_1, ..., \mu_n) \in \mathbb R^n$ and $$\Sigma = \left( \begin{array}{ccc} E[(X_1-\mu_1)(X_1-\mu_1)] & \cdots & E[(X_1-\mu_1)(X_n-\mu_n)] \\ \vdots & \ddots & \vdots \\ E[(X_n-\mu_n)(X_1-\mu_1)] & \cdots & E[(X_n-\mu_n)(X_n-\mu_n)] \end{array} \right)$$

$\;$

is the covariance matrix. I want to differentiate $D$ with respect to $\boldsymbol\mu$ and $\Sigma$. Can someone show me how to do this? In other words, how to calculate:

$$\frac{\partial D}{\partial \boldsymbol \mu} \;\;\text{and}\;\;\frac{\partial D}{\partial \Sigma}$$? Thnx for any help!

I got the motivation for my question from this source (page 13, EM-algorithm):

http://ptgmedia.pearsoncmg.com/images/0131478249/samplechapter/0131478249_ch03.pdf

$\endgroup$
4
$\begingroup$

For convenience, define the variables $$\eqalign{ \boldsymbol{z} &= \boldsymbol{x-\mu} \cr \boldsymbol{B} &= \boldsymbol{\Sigma}^{-1} \cr } $$

and note their differentials $$\eqalign{ \boldsymbol{dz} &= \boldsymbol{dx = -d\mu} \cr \boldsymbol{dB} &= \boldsymbol{-B \cdot dB^{-1} \cdot B} \cr &= \boldsymbol{-B \cdot d\Sigma \cdot B} \cr } $$ $$ $$ Next, re-cast your objective function (taking advantage of the symmetry of $\boldsymbol B$) in terms of these variables $$\eqalign{ D &= \boldsymbol{B:zz} \cr dD &= \boldsymbol{dB:zz + 2B:z\,dz} \cr &= \boldsymbol{zz:dB + 2(B\cdot z)\cdot dz} \cr } $$

and take derivatives $$\eqalign{ \frac{\partial D}{\partial \boldsymbol z} &= \boldsymbol{0 + 2(B\cdot z)} \cr \cr \frac{\partial D}{\partial \boldsymbol B} &= \boldsymbol{zz + 0} \cr } $$ $$ $$ Now use the chain rule to revert to the original variables.

For $\boldsymbol\mu$ we have $$\eqalign{ dD &= \frac{\partial D}{\partial \boldsymbol z}\cdot \boldsymbol{dz} \cr &= \boldsymbol{2(B\cdot z)\cdot (-d\mu)} \cr \cr \frac{\partial D}{\partial \boldsymbol \mu} &= \boldsymbol{-2(B\cdot z)} \cr &= \boldsymbol{-2\Sigma^{-1}\cdot (x-\mu)} \cr } $$ $$ $$ And for $\boldsymbol\Sigma$ $$\eqalign{ dD &= \frac{\partial D}{\partial \boldsymbol B}: \boldsymbol{dB} \cr &= \boldsymbol{zz:(-B\cdot d\Sigma\cdot B)} \cr &= \boldsymbol{(-B\cdot zz\cdot B):(d\Sigma)} \cr \cr \frac{\partial D}{\partial \boldsymbol \Sigma} &= \boldsymbol{-B\cdot zz\cdot B} \cr &= \boldsymbol{-\Sigma^{-1}\cdot (x-\mu)(x-\mu)\cdot \Sigma^{-1}} \cr } $$

$\endgroup$
  • $\begingroup$ +1 Thank you for your help =) Could you perhaps elaborate more on why $dz = dx = -d\mu$ and $dB = -B\cdot d\Sigma\cdot B$? Also what does $:$ mean? =) is it division? $\endgroup$ – jjepsuomi Jan 1 '14 at 21:05
  • 1
    $\begingroup$ $\boldsymbol{A:B}$ in index notation is $A_{ik}B_{ik}$; it can also be expressed as $\rm tr(\boldsymbol{A\cdot B^T})$. Given the change-of-variable $\boldsymbol{z \equiv x - y}$, if you hold $\boldsymbol{y}$ constant and allow $\boldsymbol{x}$ to vary, then $\boldsymbol{dz = dx}$. If you take $\boldsymbol{x}$ as constant, then $\boldsymbol{dz = -dy}$ The formula for $\boldsymbol{dB = d\Sigma^{-1}}$ is well known; it can be derived by differentiating the equation $\boldsymbol{\Sigma\cdot \Sigma^{-1} = I}$. $\endgroup$ – lynne Jan 2 '14 at 0:23
  • $\begingroup$ +1 @lynne Thank you very much =) One last thing to wrap this up, could you give me a link etc. to a source where I can study about this formula $dB = d\Sigma^{-1}$ =) $\endgroup$ – jjepsuomi Jan 2 '14 at 5:58
  • 1
    $\begingroup$ en.wikipedia.org/wiki/… $\endgroup$ – lynne Jan 2 '14 at 6:27
  • $\begingroup$ @lynne I think that if you expand your answer with full details, it could be a very useful resource (and will be heavily upvoted).. $\endgroup$ – MadHatter Jun 3 '17 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.