I am reading this: http://www.cs.cmu.edu/~tom/mlbook/Joint_MLE_MAP.pdf

I found it extremely useful. However, I am a bit confused about the difference between $P(D|\theta)$ and $P(\theta|D)$.

The document first declares D as a sequence of $n$ consecutive coin tosses each of which is independent of each other and $\theta$ is the probability of, say, heads, that we are trying to predict. (For example, D=1010001001)

Then, it says Maximum Likelihood Estimation (MLE) defines $\theta$ by,

$$ \theta^{\mathrm{MLE}} = \underset{\theta} {\mathrm{argmax}} ~ P(D~|~\theta). $$

If we see $\alpha_0$ heads and $\alpha_1$ tails in D ($\alpha_0 + \alpha_1 = n$), we should maximize,

$$ \theta^{\alpha_0}. (1-\theta)^{\alpha_1} $$

Then, document uses derivation of this and sets it to $0$ to find which $\theta$ maximizes that quantity. Until this point everything is clear on my mind.

Then, it switches to Maximum a Posteriori Probability Estimation (MAP). And it says we should maximize $P(\theta~|~D)$ for MAP, which is,

$$ \theta^{\mathrm{MAP}} = \underset{\theta} {\mathrm{argmax}} ~ P(\theta~|~D). $$

and it uses Bayes formula to translate $P(\theta~|~D)$ to $P(D~|~\theta)$. In the process of conversion $P(\theta~|~D)$ to $P(D~|~\theta)$, we also get $P(D)$ and $P(\theta)$,

$$ \theta^{\mathrm{MAP}} = \underset{\theta} {\mathrm{argmax}} ~ P(\theta~|~D) = \underset{\theta} {\mathrm{argmax}} ~ \frac{ P(D~|~\theta) P(\theta) }{P(D)}. $$

My questions are:

  1. I can understand $P(D~|~\theta) = \theta^{\alpha_0}. (1-\theta)^{\alpha_1}$ in this example. However, I am having hard time to grasp what does $P(\theta~|~D)$ mean conceptually in this example.
  2. What does $P(\theta)$ and $P(D)$ mean in the last equation in this concept?

In the above say that you believe that $\theta$ lies somewhere between $0 and 1$ but you have no reason to favor one value over the other (before observing outcomes). Then you can start with the assumption that $\theta$ is a uniform random variable on $[0, 1]$.

  1. $P(\theta | D)$ means: if you now observe the outcome $D$, what is the probability that $D$ was chosen from the distribution with parameter $\theta$.

  2. $P(\theta)$ is given by the uniform distribution (by assumption) and you can calculate $P(D)$ by: $$P(D) = \int_0^1 P(D | \theta) d\theta$$ That quantity is the probability you observe $D$ given your assumption on the distribution of $\theta$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.