Differentiating mahalanobis distance I would like to differentiate the mahalanobis distance:
$$D(\textbf{x}, \boldsymbol \mu, \Sigma) = (\textbf{x}-\boldsymbol \mu)^T\Sigma^{-1}(\textbf{x}-\boldsymbol \mu)$$
where $\textbf{x} = (x_1, ..., x_n) \in \mathbb R^n, \;\boldsymbol \mu = (\mu_1, ..., \mu_n) \in \mathbb R^n$ and 
$$\Sigma = \left( \begin{array}{ccc}
E[(X_1-\mu_1)(X_1-\mu_1)] & \cdots & E[(X_1-\mu_1)(X_n-\mu_n)] \\
\vdots & \ddots & \vdots \\
E[(X_n-\mu_n)(X_1-\mu_1)] & \cdots & E[(X_n-\mu_n)(X_n-\mu_n)] \end{array} \right)$$ 
$\;$
is the covariance matrix. I want to differentiate $D$ with respect to $\boldsymbol\mu$ and $\Sigma$. Can someone show me how to do this? In other words, how to calculate:
$$\frac{\partial D}{\partial \boldsymbol \mu} \;\;\text{and}\;\;\frac{\partial D}{\partial \Sigma}$$? Thnx for any help!
I got the motivation for my question from this source (page 13, EM-algorithm):
http://ptgmedia.pearsoncmg.com/images/0131478249/samplechapter/0131478249_ch03.pdf
 A: For convenience, define the variables
$$\eqalign{
\boldsymbol{z} &= \boldsymbol{x-\mu} \cr
\boldsymbol{B} &= \boldsymbol{\Sigma}^{-1} \cr
} $$
and note their differentials
$$\eqalign{
\boldsymbol{dz} &= \boldsymbol{dx = -d\mu} \cr
\boldsymbol{dB} &= \boldsymbol{-B \cdot dB^{-1} \cdot B} \cr
                &= \boldsymbol{-B \cdot d\Sigma \cdot B} \cr
} $$
$$ $$
Next, re-cast your objective function (taking advantage of the symmetry of $\boldsymbol B$) in terms of these variables
$$\eqalign{
D &= \boldsymbol{B:zz} \cr
dD &= \boldsymbol{dB:zz + 2B:z\,dz} \cr
   &= \boldsymbol{zz:dB + 2(B\cdot z)\cdot dz} \cr
} $$
and take derivatives
$$\eqalign{
\frac{\partial D}{\partial \boldsymbol z} &= \boldsymbol{0 + 2(B\cdot z)} \cr
  \cr
\frac{\partial D}{\partial \boldsymbol B} &= \boldsymbol{zz + 0} \cr
} $$
$$ $$
Now use the chain rule to revert to the original variables.
For $\boldsymbol\mu$ we have
$$\eqalign{
dD &= \frac{\partial D}{\partial \boldsymbol z}\cdot \boldsymbol{dz} \cr
   &= \boldsymbol{2(B\cdot z)\cdot (-d\mu)} \cr
  \cr
\frac{\partial D}{\partial \boldsymbol \mu} &= \boldsymbol{-2(B\cdot z)} \cr
   &= \boldsymbol{-2\Sigma^{-1}\cdot (x-\mu)} \cr
} $$
$$ $$
And for $\boldsymbol\Sigma$ 
$$\eqalign{
dD &= \frac{\partial D}{\partial \boldsymbol B}: \boldsymbol{dB} \cr
   &= \boldsymbol{zz:(-B\cdot d\Sigma\cdot B)} \cr
   &= \boldsymbol{(-B\cdot zz\cdot B):(d\Sigma)} \cr
  \cr
\frac{\partial D}{\partial \boldsymbol \Sigma} &= \boldsymbol{-B\cdot zz\cdot B} \cr
   &= \boldsymbol{-\Sigma^{-1}\cdot (x-\mu)(x-\mu)\cdot \Sigma^{-1}} \cr
} $$
A: We have:
$$\eqalign{
D&=(x-\mu)^T\Sigma^{-1}(x-\mu) \cr
\Sigma&=\Sigma^T \cr
D&\in\mathbb{R} \cr
x,\mu&\in\mathbb{R}^N \cr
\Sigma&\in\mathbb{R}^{N\times N} \cr
} $$
We want to find $\frac{\partial D}{\partial \mu}\in\mathbb{R}^N$ and $\frac{\partial D}{\partial \Sigma}\in\mathbb{R}^{N\times N}$, with:
$$\eqalign{
\left(\frac{\partial D}{\partial \mu}\right)_i&=\frac{\partial D}{\partial \mu_i} \cr
\left(\frac{\partial D}{\partial \Sigma}\right)_{ij}&=\frac{\partial D}{\partial \Sigma_{ij}} \cr
} $$
Following lynne's answer, we'll define:
$$\eqalign{
z&=x-\mu\in\mathbb{R}^N \cr
B&=\Sigma^{-1}\in\mathbb{R}^{N\times N} \cr
} $$
So $D$ simplifies to $z^TBz$.
We'll start by finding $\frac{\partial D}{\partial \mu}$, by using the chain-rule and expanding the matrix-vector multiplications, and by noting that $\frac{\partial z_i}{\partial \mu_i}=-1$:
$$\eqalign{
\frac{\partial D}{\partial \mu_i}&=\frac{\partial D}{\partial z_i}\frac{\partial z_i}{\partial \mu_i} \cr
&=-\frac{\partial D}{\partial z_i} \cr
&=-\frac{\partial}{\partial z_i}\left[z^TBz\right] \cr
&=-\frac{\partial}{\partial z_i}\left[\sum_{j,k}{\left[B_{jk}z_jz_k\right]}\right] \cr
&=-\frac{\partial}{\partial z_i}\left[\sum_{j\ne i,k\ne i}{\left[B_{jk}z_jz_k\right]}+\sum_{j\ne i}{\left[B_{ji}z_jz_i\right]}+\sum_{k\ne i}{\left[B_{ik}z_iz_k\right]}+B_{ii}z_i^2\right] \cr
&=-\sum_{j\ne i}{\left[B_{ji}z_j\right]}-\sum_{k\ne i}{\left[B_{ik}z_k\right]}-2B_{ii}z_i \cr
&=-2\sum_{j}{\left[B_{ij}z_j\right]} \cr
&=-2(Bz)_i \cr
\Rightarrow\frac{\partial D}{\partial \mu}&=-2Bz \cr
&=-2\Sigma^{-1}(x-\mu) \cr
} $$
To find $\frac{\partial D}{\partial \Sigma}$, we'll start off by noting that the partial derivative $\frac{\partial D}{\partial A}$ of the scalar $D$ with respect to any matrix $A$ can be expressed in terms of the total differential of a small change $dD$ in $D$ with respect to a small change $dA$ in the matrix $A$, which we can express in terms of the trace operator as follows:
$$\eqalign{
dD&=\sum_{i,j}{\left[\frac{\partial D}{\partial A_{ij}}dA_{ij}\right]} \cr
&=\sum_{i,j}{\left[\left(\frac{\partial D}{\partial A}\right)_{ij}dA_{ij}\right]} \cr
&=\sum_{i,j}{\left[\left(\frac{\partial D}{\partial A}\right)^T_{ji}dA_{ij}\right]} \cr
&=\sum_{j}{\left[\left(\frac{\partial D}{\partial A}^TdA\right)_{jj}\right]} \cr
&=\mathrm{Tr}\left(\frac{\partial D}{\partial A}^TdA\right) \cr
} $$
We'll now introduce the notation $A:B=\mathrm{Tr}(A^TB)$ (note that the $:$ operator can be thought of like a dot-product between two matrices, which takes two matrices as input and returns a scalar), and note that $\mathrm{Tr}(AB)=\sum_{i}{\left[(AB)_{ii}\right]}=\sum_{i,j}{\left[(A_{ij}B_{ji})\right]}=\sum_{i,j}{\left[B_{ji}A_{ij}\right]}=\sum_{j}{\left[(BA)_{jj}\right]}=\mathrm{Tr}(BA)$. Therefore we have, for any matrix $A$:
$$\eqalign{
dD&=\frac{\partial D}{\partial A}:dA \cr
} $$
Returning to $\frac{\partial D}{\partial \Sigma}$, which we will start off by expressing in terms of $B=\Sigma^{-1}$:
$$\eqalign{
dD&=\frac{\partial D}{\partial B}:dB \cr
\left(\frac{\partial D}{\partial B}\right)_{ij}&=\frac{\partial}{\partial B_{ij}}\left[z^TBz\right] \cr
&=\frac{\partial}{\partial B_{ij}}\left[\sum_{k,l}{B_{kl}z_kz_l}\right] \cr
&=z_iz_j \cr
&=(zz^T)_{ij} \cr
\Rightarrow \frac{\partial D}{\partial B}&=zz^T \cr
\Rightarrow dD&=zz^T:dB \cr
&=zz^T:d\Sigma^{-1} \cr
} $$
Now we will express a small change $d\Sigma^{-1}$ in $\Sigma^{-1}$ in terms of a small change $d\Sigma$ in $\Sigma$:
$$\eqalign{
d\Sigma^{-1}&=(\Sigma+d\Sigma)^{-1}-\Sigma^{-1} \cr
&=\left((\Sigma+d\Sigma)^{-1}\cdot\Sigma\cdot\Sigma^{-1}\right)-\left((\Sigma+d\Sigma)^{-1}\cdot(\Sigma+d\Sigma)\cdot\Sigma^{-1}\right) \cr
&=(\Sigma+d\Sigma)^{-1}\cdot\left(\Sigma-(\Sigma+d\Sigma)\right)\cdot\Sigma^{-1} \cr
&=-(\Sigma+d\Sigma)^{-1}\cdot d\Sigma\cdot\Sigma^{-1} \cr
&=-\Sigma^{-1}\cdot d\Sigma\cdot\Sigma^{-1} \cr
} $$
Where the last step comes from the fact that $d\Sigma$ is small relative to $\Sigma$. Finally, returning to $\frac{\partial D}{\partial \Sigma}$, we have:
$$\eqalign{
dD&=zz^T:d\Sigma^{-1} \cr
&=zz^T:\left(-\Sigma^{-1}\cdot d\Sigma\cdot\Sigma^{-1}\right) \cr
&=\mathrm{Tr}\left((zz^T)^T\left(-\Sigma^{-1}\cdot d\Sigma\cdot\Sigma^{-1}\right)\right) \cr
&=-\mathrm{Tr}\left(z\cdot z^T\cdot\Sigma^{-1}\cdot d\Sigma\cdot\Sigma^{-1}\right) \cr
&=-\mathrm{Tr}\left(\left(z\cdot z^T\cdot\Sigma^{-1}\cdot d\Sigma\right)\cdot\left(\Sigma^{-1}\right)\right) \cr
&=-\mathrm{Tr}\left(\left(\Sigma^{-1}\right)\cdot\left(z\cdot z^T\cdot\Sigma^{-1}\cdot d\Sigma\right)\right) \cr
&=-\mathrm{Tr}\left(\left(\Sigma^{-1}\cdot z\cdot z^T\cdot\Sigma^{-1}\right)\cdot\left(d\Sigma\right)\right) \cr
&=-\left(\Sigma^{-1}zz^T\Sigma^{-1}\right):d\Sigma \cr
&=\frac{\partial D}{\partial \Sigma}:d\Sigma \cr
\Rightarrow\frac{\partial D}{\partial \Sigma}&=-\Sigma^{-1}zz^T\Sigma^{-1} \cr
&=-\left(\Sigma^{-1}z\right)\left(\Sigma^{-1}z\right)^T \cr
} $$
