Let $\mathbf{x}\in\Bbb{R}^n$ be an $n$-dimensional real vector distributed normally with mean vector $\mu\in\Bbb{R}^n$ and covariance matrix $\Sigma$; i.e. $\mathbf{x}\sim\mathcal{N}(\mu,\Sigma)$.

The Mahalanobis distance of $\mathbf{x}$ is defined as $$ D_M(\mathbf{x}) = \sqrt{ (\mathbf{x}-\mu)^\top\Sigma^{-1}(\mathbf{x}-\mu) }. $$

I am interested in defining a distance between two $n$-dimensional random vectors $\mathbf{x}\sim\mathcal{N}(\mu_x,\Sigma_x)$ and $\mathbf{y}\sim\mathcal{N}(\mu_y,\Sigma_y)$ in a similar way as above.

The Bhattacharyya distance between two multivariate distributions $\mathcal{N}(\mu_x,\Sigma_x)$ and $\mathcal{N}(\mu_y,\Sigma_y)$ includes the following "Mahalanobis term" (as Wikipedia suggests): $$ d(\mathbf{x},\mathbf{y})=\sqrt{(\mathbf{x}-\mathbf{y})^\top \Bigg(\frac{\Sigma_x+\Sigma_y}{2}\Bigg)^{-1}(\mathbf{x}-\mathbf{y})}. $$

Would that be a good candidate?


Yes, that is a fine candidate, because in the first case you are basically computing the distance to the mean, but for two distributions you have two actual sample vectors you can compute the distance between, so there's really no reason to include the means in the distance calculation. But if you're interested in variants, the basic issue is how to define the matrix in the middle. If $x$ and $y$ were sampled from distributions with different covariances $\Sigma_x$ and $\Sigma_y$, then one natural thing to do is to take the average of the two covariances, as you have found. At the very least, this is consistent in the sense that you get $\Sigma_x^{-1}$ as the inverse matrix if $\Sigma_x = \Sigma_y$. However there are possibly variants which may, for example, take the average of the inverses rather than the inverse of the average. This also has the consistency property when $\Sigma_x = \Sigma_y$. I would have to read more to explain perhaps why they chose the average of the covariances (before taking inverse) over all other choices

| cite | improve this answer | |
  • $\begingroup$ Thank you very much for your answer. I wonder if it is also alright to take the mean vectors as follows: $$ (\mu_x-\mu_y)^\top\Bigg(\frac{\Sigma_x+\Sigma_y}{2}\Bigg)^{-1}(\mu_x-\mu_y). $$ $\endgroup$ – nullgeppetto Jul 13 '15 at 8:49
  • $\begingroup$ Also, I cannot find a good motivation for that distance choice, what do you think about that? $\endgroup$ – nullgeppetto Jul 13 '15 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.