0
$\begingroup$

I'm asked to compare the distance of the approximated distribution and the true distribution in a Bayesian approach. While we used Laplace approximation to find the MAP of the target posterior distribution, one paper reviewer asked us to estimate the distance of our approximated distribution to the true posterior.

I'm confused about it. We meant to using LA to calculate an approximated MAP, so we only care about the argmax of the posterior. In our problem, the dimensionality is high and apparently a Gaussian approximation is very different to the true distribution. The overall distance will be high. So what's the point to compare the distance of the whole distribution? Plus I've been reading some machine learning papers and I haven't seen any graphical model methods, or Bayesian approaches comparing their approximated inference to an intractable exact inference.

Could anybody give me some idea about this problem? May I have a reference to see how we shall address this issue? Thanks!

$\endgroup$
  • $\begingroup$ so you wanna measure the divergence between two distributions and you need a metric for that? $\endgroup$ – Seyhmus Güngören Feb 16 '14 at 0:27
  • $\begingroup$ Thanks. Yes, actually I've been tried to use an approximated KL-divergence as a metric but not sure how I should explain these values. (They are just like 3.5e+03). It would be really helpful if there are some reference papers that I can see how other people address this issue. $\endgroup$ – klu Feb 16 '14 at 0:38
0
$\begingroup$

Okay I suggest you to use the Hellinger distance or the squared Hellinger distance. It is symmetric and scales in $[0,1]$. You can simply talk about a certain percentage of closeness, then. Check wiki for details. If you want to have a complete list of distances you can find it here

$\endgroup$
  • $\begingroup$ Thanks for the keyword. This metric actually helps. However, I expect I will get a value close to 1, that means the distance is very far. Because the approximated distribution is a simple Gaussian while the true posterior is a complicated high dimensional distribution, it makes sense to be very different. While I approximate it only for the MAP, I think the overall distance really doesn't matter. Do you have any suggestion that I could make a convincing statement on this point? Thanks very much! $\endgroup$ – klu Feb 16 '14 at 10:26
  • $\begingroup$ Isnt your true Gaussian distribution also high-dimensional? It is usually very difficult to get $0$ by Hellinger distance. $\endgroup$ – Seyhmus Güngören Feb 16 '14 at 13:21
  • $\begingroup$ The true distribution isn't Gaussian. It's a complicated joint distribution. So I actually used a Monte Carlo integration to calculate this distance. Their dimension are the same. $\endgroup$ – klu Feb 16 '14 at 18:26
  • $\begingroup$ No matter what distribution you have as long as they are of same dimensionality you can calculate the Hellinger distance. You can also argue why it makes sense to have different distributions. $\endgroup$ – Seyhmus Güngören Feb 16 '14 at 18:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.