In Bayesian approaches, comparing the approximated distribution and the true distribution

I'm asked to compare the distance of the approximated distribution and the true distribution in a Bayesian approach. While we used Laplace approximation to find the MAP of the target posterior distribution, one paper reviewer asked us to estimate the distance of our approximated distribution to the true posterior.

I'm confused about it. We meant to using LA to calculate an approximated MAP, so we only care about the argmax of the posterior. In our problem, the dimensionality is high and apparently a Gaussian approximation is very different to the true distribution. The overall distance will be high. So what's the point to compare the distance of the whole distribution? Plus I've been reading some machine learning papers and I haven't seen any graphical model methods, or Bayesian approaches comparing their approximated inference to an intractable exact inference.

Could anybody give me some idea about this problem? May I have a reference to see how we shall address this issue? Thanks!

• so you wanna measure the divergence between two distributions and you need a metric for that? – Seyhmus Güngören Feb 16 '14 at 0:27
• Thanks. Yes, actually I've been tried to use an approximated KL-divergence as a metric but not sure how I should explain these values. (They are just like 3.5e+03). It would be really helpful if there are some reference papers that I can see how other people address this issue. – klu Feb 16 '14 at 0:38

Okay I suggest you to use the Hellinger distance or the squared Hellinger distance. It is symmetric and scales in $[0,1]$. You can simply talk about a certain percentage of closeness, then. Check wiki for details. If you want to have a complete list of distances you can find it here
• Isnt your true Gaussian distribution also high-dimensional? It is usually very difficult to get $0$ by Hellinger distance. – Seyhmus Güngören Feb 16 '14 at 13:21