Probability Distributions : "Most Likely" vs "Most Common"? Consider the problem of Bayesian Estimation - that is, suppose we are interested in estimating the probability distribution(i.e. posterior distribution) parameter "thetha" based on some observed data and some prior knowledge of
"thetha".
We can write the Posterior Distribution as follows:
$$P(\theta \mid \mathcal{D}) = \frac{P(\mathcal{D} \mid \theta) P(\theta)}{P(\mathcal{D})}$$
Where:

*

*$P(\theta \mid \mathcal{D})$ is the posterior distribution, which represents our updated belief about the parameter $\theta$ given the data $\mathcal{D}$.


*$P(\mathcal{D} \mid \theta)$ is the likelihood function, which represents the probability of observing the data $\mathcal{D}$ given the parameter $\theta$.


*$P(\theta)$ is the prior distribution, which represents our initial belief about the parameter $\theta$ before observing any data.


*$P(\mathcal{D})$ is the likelihood, which is the probability of observing the data $\mathcal{D}$, averaged over all possible values of the parameter $\theta$
Once we have derived an expression for this Posterior Distribution - we are usually interested in one of two things:
1) "Most Likely Point" : This is usually referred to as the "Bayes Estimator" - in practice, this refers to the Expected Value of the Posterior Distribution. Note that in many applications, we can not analytically evaluate this expectation and are forced to use some probabilistic sampling algorithm (e.g. MCMC) to take random samples and then average these samples as an estimate:
$$\hat{\theta}_{\mathrm{Bayes}} = \mathbb{E}[\theta \mid \mathcal{D}] = \int \theta P(\theta \mid \mathcal{D}) \mathrm{d}\theta$$
2) "Most Common Point": This is usually referred to the "Maximum A Posteriori Estimate" (MAP Estimate) and is analogous to the "Mode" of the Probability Distribution. I heard that we can calculate the MAP by setting the derivative of the Posterior Distribution to 0 and then solving this equation numerically (e.g. using BFGS):
$$\hat{\theta}{\mathrm{MAP}} = \operatorname*{argmax} P(\theta \mid \mathcal{D})  = \left.\frac{d}{d\theta} \log P(\theta \mid \mathcal{D})\right|{\theta = \hat{\theta}_{\mathrm{MAP}}} = 0$$
My Question: From a conceptual standpoint, what is the difference between the "Most Likely Point" and the "Most Common Point" in a Probability Distribution? In what kinds of situations is it more informative to work with the "Most Likely Point" vs the "Most Common Point"? Do these have different mathematical properties?
Thanks!
 A: According to your definitions, "most likely" means expected value (of posterior distribution), while "most common" means maximum (of posterior distribution). Both estimates are possible, but the first is usually used in Bayesian inference.
A: I'm not a huge fan of the phrasing "Most likely point", since the expected value is not always very likely - rather I would call it something like "The average point".
This more aptly illustrates the difference between the estimators in that one is based on the average value of the parameter to be estimated and hence is appropriate when the (conditional) distribution is relatively close to its mean, i.e. when the variance is low.
Meanwhile, if the variance is high, this implies that a lot of the data may be far away from mean, and so the Bayes estimator is not appropriate, and it might be better to use the "Most common point".
To see this, let's look at a concrete toy example.
Suppose that the parameter $\theta$, given data $\mathcal D$ is either $0$ or $1$, i.e. $\theta\mid \mathcal D\sim \mathrm{Bernoulli}(p)$ for some $p\in[0, 1]$.
Then the Bayes estimator is $\widehat{\theta}_B\mid \mathcal D=\mathbb{E}[\theta\mid \mathcal D]=p$, and the MAP estimator is $\widehat{\theta}_{\mathrm{MAP}}=1_{\{p>1/2\}}$ (since it is $1$ if this is the most likely outcome and $0$ otherwise).
To compare these estimators, let us use the mean square error.
The mean squared error of the Bayes estimator is just the conditional variance, i.e.
$$
 \mathrm{MSE}(\widehat{\theta}_B\mid \mathcal D)
 =\mathbb{E}[(\widehat{\theta}_B-\theta)^2\mid \mathcal D]
 =\mathbb{V}[\theta\mid \mathcal D]
 =p(1-p).
$$
Meanwhile, the MSE of the MAP estimator is
\begin{align*}
 \mathrm{MSE}(\widehat{\theta}_{\mathrm{MAP}}\mid \mathcal D)
 &=\mathbb{E}[(\widehat{\theta}_{\mathrm{MAP}}-\theta)^2\mid \mathcal D] \\
 &=(1-p)1_{\{p>1/2\}}+p(1_{\{p>1/2\}}-1)^2 \\
 &=(1-p)1_{\{p>1/2\}}+p1_{\{p\le 1/2\}} \\
 &=\min\{p, 1-p\}.
\end{align*}
As we see, the MSE of the Bayes estimator is the highest when the variance is high, i.e. when $p\approx0$ or $p\approx1$, which is exactly when the MAP estimator performs the best.
