0
$\begingroup$

The prior has a form such that it is $0.5$ for $\theta=0.6$ or $\theta=0.2$ and $0$ elsewhere. The likelihood function $P(D|h)$ has Bernoulli form.

Hence, posterior is $0.5P(D|h)$ for $\theta=0.6$ or $\theta=0.2$ and $0$ elsewhere.

When I calculate the MAP estimate, it comes out to be independent of theta as $N_1/N$ which is same value as obtained from uniform prior, which is strange.

If we change prior such that it is $0.4$ for $\theta=0.6$ and $0.6$ for $\theta=0.2$ and $0$ elsewhere, the MAP estimate still comes out to be same.

Is it correct to say that such discrete prior has no effect on MAP estimate?

$\endgroup$
5
  • $\begingroup$ I do not understand. The MAP (i.e. the mode of the posterior distribution) must lie in the support of the prior distribution, which in this case seems to be $\{0.2,0.6\}$. So here it cannot be $\frac{N_1}{N}$ unless that is $0.2$ or $0.6$. Do you mean MLE rather than MAP? $\endgroup$
    – Henry
    Commented Jan 28, 2022 at 0:49
  • $\begingroup$ I mean MAP and I am trying to understand what happens if we have prior as {0.2,0.6}. When I solve for the given set of values, my MAP estimate comes to be same as MLE estimate. $\endgroup$
    – NKR
    Commented Jan 28, 2022 at 1:36
  • $\begingroup$ See what happens when $N_1=9$ and $N=23$ with your two priors so $\frac{N_1}{N}\approx 0.39 \not \in \{0.2,0.6\}$. Note that if your interpretation of MLE is restricted to choosing between $0.2$ and $0.6$ then giving a prior probability of $50\%$ to each is in effect a uniform prior on those two values $\endgroup$
    – Henry
    Commented Jan 28, 2022 at 2:18
  • $\begingroup$ @Henry That is correct. However, when we actually calculate the MAP estimate by taking derivative, it becomes independent of given prior ({0.2,0.6}).. which is exactly my concern. $\endgroup$
    – NKR
    Commented Jan 29, 2022 at 23:27
  • $\begingroup$ @NKR If the prior only has two-point support, the MAP chooses the greater posterior between those two points. No derivatives (pls see my updated reply). $\endgroup$ Commented Jan 29, 2022 at 23:42

1 Answer 1

0
$\begingroup$

A MAP estimator maximizes

$$\underbrace{f(\theta|x)}_{\text{posterior}}\propto \underbrace{f(x|\theta)}_{\text{likelihood}}\underbrace{f(\theta)}_{\text{prior}}$$

with respect to $\theta$. In general, the MAP estimator will depend on both the likelihood and the prior since both depend on $\theta$. In the special case of a uniform prior, $f(\theta)$ is constant, so the MAP estimator coincides with the maximum likelihood estimator (MLE) (assuming the prior contains the MLE in its support).


In your setup, if your data $x_i$ is iid Bernoulli($\theta$), and your prior for $\theta$ has two-point support $\{\theta_1,\theta_2\}$ with respective hyperparameter masses $p,1-p\in (0,1)$ then for $x_i\in \{0,1\},$

$$f(x|\theta)=\Pi_i\theta^x_i (1-\theta)^{1-x_i}=\theta^{\sum_i x_i}(1-\theta)^{n-\sum_i x_i},\\ f(\theta)=p^{\frac{\theta-\theta_2}{\theta_1-\theta_2}}(1-p)^{\frac{\theta-\theta_1}{\theta_2-\theta_1}}\bf{1}_{\theta\in \{\theta_1,\theta_2\}}.$$

The log posterior for $\theta\in\{\theta_1,\theta_2\}$ is

$$\small \log f(\theta|x)=\text{const}+\sum_i x_i\log \theta+(n-\sum_i x_i)\log (1-\theta)+\frac{\theta-\theta_2}{\theta_1-\theta_2}\log p +\frac{\theta-\theta_1}{\theta_2-\theta_1}\log (1-p)$$

So to find the MAP, you just have to check which of $\theta_1,\theta_2$ gives a higher log posterior. Equivalently, letting $\bar x:=\frac{1}{n}\sum_i x_i,$

$$\small \hat \theta_{\text{MAP}}={\arg\max}_{\theta\in\{\theta_1,\theta_2\}}\left\{ \bar x\log \theta+(1-\bar x)\log (1-\theta)+\left(\frac{1}{n}\log p\right){\bf 1}_{\theta=\theta_1}+\left(\frac{1}{n}\log (1-p)\right){\bf 1}_{\theta=\theta_2}\right\}.$$

The estimator depends on data and prior hyperparameters ($\theta_1,\theta_2,p$). But it is possible that for a particular data set, a small change in $p$ will not change the MAP estimator.

$\endgroup$
11
  • $\begingroup$ In general, the MAP estimator depends on both likelihood and the prior, since both are functions of $\theta$ $\endgroup$ Commented Jan 28, 2022 at 1:15
  • $\begingroup$ Thanks for your answer. For my case, p=0.5. This means third term in FOC would become 0. So it seems that we would get the same value of 𝜃 as with uniform prior. Please me know if thats correct. $\endgroup$
    – NKR
    Commented Jan 28, 2022 at 1:32
  • $\begingroup$ True, the third term is zero if $p=0.5$ (this is a discrete uniform prior). More generally, the MAP coincides with MLE under a uniform prior (assuming the prior contains the MLE in its support). But it is incorrect to say any discrete prior has no effect on the MAP estimate; in your setup, you can see the third term in the FOC is strictly monotonic in $p$ so it varies uniquely with the masses assigned to your two-point prior (e.g. it would not be zero if $p=0.4$ as you mentioned in your question). $\endgroup$ Commented Jan 28, 2022 at 1:57
  • $\begingroup$ To make sure I understood it correctly, for prior with two-point support {x1, x2} with p=0.5, MAP coincides with MLE (with bernoulli dist). However, with any other value of p, this might not be true. $\endgroup$
    – NKR
    Commented Jan 28, 2022 at 2:07
  • $\begingroup$ Yes, do you see why it is not true for other values of $p$ though? $\endgroup$ Commented Jan 28, 2022 at 2:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .