# MAP estimate for a discrete prior

The prior has a form such that it is $$0.5$$ for $$\theta=0.6$$ or $$\theta=0.2$$ and $$0$$ elsewhere. The likelihood function $$P(D|h)$$ has Bernoulli form.

Hence, posterior is $$0.5P(D|h)$$ for $$\theta=0.6$$ or $$\theta=0.2$$ and $$0$$ elsewhere.

When I calculate the MAP estimate, it comes out to be independent of theta as $$N_1/N$$ which is same value as obtained from uniform prior, which is strange.

If we change prior such that it is $$0.4$$ for $$\theta=0.6$$ and $$0.6$$ for $$\theta=0.2$$ and $$0$$ elsewhere, the MAP estimate still comes out to be same.

Is it correct to say that such discrete prior has no effect on MAP estimate?

• I do not understand. The MAP (i.e. the mode of the posterior distribution) must lie in the support of the prior distribution, which in this case seems to be $\{0.2,0.6\}$. So here it cannot be $\frac{N_1}{N}$ unless that is $0.2$ or $0.6$. Do you mean MLE rather than MAP? Commented Jan 28, 2022 at 0:49
• I mean MAP and I am trying to understand what happens if we have prior as {0.2,0.6}. When I solve for the given set of values, my MAP estimate comes to be same as MLE estimate.
– NKR
Commented Jan 28, 2022 at 1:36
• See what happens when $N_1=9$ and $N=23$ with your two priors so $\frac{N_1}{N}\approx 0.39 \not \in \{0.2,0.6\}$. Note that if your interpretation of MLE is restricted to choosing between $0.2$ and $0.6$ then giving a prior probability of $50\%$ to each is in effect a uniform prior on those two values Commented Jan 28, 2022 at 2:18
• @Henry That is correct. However, when we actually calculate the MAP estimate by taking derivative, it becomes independent of given prior ({0.2,0.6}).. which is exactly my concern.
– NKR
Commented Jan 29, 2022 at 23:27
• @NKR If the prior only has two-point support, the MAP chooses the greater posterior between those two points. No derivatives (pls see my updated reply). Commented Jan 29, 2022 at 23:42

A MAP estimator maximizes

$$\underbrace{f(\theta|x)}_{\text{posterior}}\propto \underbrace{f(x|\theta)}_{\text{likelihood}}\underbrace{f(\theta)}_{\text{prior}}$$

with respect to $$\theta$$. In general, the MAP estimator will depend on both the likelihood and the prior since both depend on $$\theta$$. In the special case of a uniform prior, $$f(\theta)$$ is constant, so the MAP estimator coincides with the maximum likelihood estimator (MLE) (assuming the prior contains the MLE in its support).

In your setup, if your data $$x_i$$ is iid Bernoulli($$\theta$$), and your prior for $$\theta$$ has two-point support $$\{\theta_1,\theta_2\}$$ with respective hyperparameter masses $$p,1-p\in (0,1)$$ then for $$x_i\in \{0,1\},$$

$$f(x|\theta)=\Pi_i\theta^x_i (1-\theta)^{1-x_i}=\theta^{\sum_i x_i}(1-\theta)^{n-\sum_i x_i},\\ f(\theta)=p^{\frac{\theta-\theta_2}{\theta_1-\theta_2}}(1-p)^{\frac{\theta-\theta_1}{\theta_2-\theta_1}}\bf{1}_{\theta\in \{\theta_1,\theta_2\}}.$$

The log posterior for $$\theta\in\{\theta_1,\theta_2\}$$ is

$$\small \log f(\theta|x)=\text{const}+\sum_i x_i\log \theta+(n-\sum_i x_i)\log (1-\theta)+\frac{\theta-\theta_2}{\theta_1-\theta_2}\log p +\frac{\theta-\theta_1}{\theta_2-\theta_1}\log (1-p)$$

So to find the MAP, you just have to check which of $$\theta_1,\theta_2$$ gives a higher log posterior. Equivalently, letting $$\bar x:=\frac{1}{n}\sum_i x_i,$$

$$\small \hat \theta_{\text{MAP}}={\arg\max}_{\theta\in\{\theta_1,\theta_2\}}\left\{ \bar x\log \theta+(1-\bar x)\log (1-\theta)+\left(\frac{1}{n}\log p\right){\bf 1}_{\theta=\theta_1}+\left(\frac{1}{n}\log (1-p)\right){\bf 1}_{\theta=\theta_2}\right\}.$$

The estimator depends on data and prior hyperparameters ($$\theta_1,\theta_2,p$$). But it is possible that for a particular data set, a small change in $$p$$ will not change the MAP estimator.

• In general, the MAP estimator depends on both likelihood and the prior, since both are functions of $\theta$ Commented Jan 28, 2022 at 1:15
• Thanks for your answer. For my case, p=0.5. This means third term in FOC would become 0. So it seems that we would get the same value of 𝜃 as with uniform prior. Please me know if thats correct.
– NKR
Commented Jan 28, 2022 at 1:32
• True, the third term is zero if $p=0.5$ (this is a discrete uniform prior). More generally, the MAP coincides with MLE under a uniform prior (assuming the prior contains the MLE in its support). But it is incorrect to say any discrete prior has no effect on the MAP estimate; in your setup, you can see the third term in the FOC is strictly monotonic in $p$ so it varies uniquely with the masses assigned to your two-point prior (e.g. it would not be zero if $p=0.4$ as you mentioned in your question). Commented Jan 28, 2022 at 1:57
• To make sure I understood it correctly, for prior with two-point support {x1, x2} with p=0.5, MAP coincides with MLE (with bernoulli dist). However, with any other value of p, this might not be true.
– NKR
Commented Jan 28, 2022 at 2:07
• Yes, do you see why it is not true for other values of $p$ though? Commented Jan 28, 2022 at 2:08