1
$\begingroup$

I am trying to optimize the first equation (likelihood of the dataset D) in order to obtain the second equation (Maximum likelihood estimate of the theta parameters).

The distribution is the Beta Binomial Distribution.

$N$ is the total number of coin flips.

$N_1$ is the number of heads.

$N_0$ is the number of tails.

So $N = N_1 + N_0$

The two equations I mentioned previously are:

$$p(\mathcal D\mid \theta) = \theta^{N_1}(1-\theta)^{N_0}$$

and

$$ \widehat\theta_\text{MLE} = \frac{N_1}{N} $$

The ^ symbol above theta means "estimate", and theta in the second equation represents the Maximum Likelihood Estimate. The fancy D in the first equation represents the set of trials (coin flips) from which we get $N$, $N_1$, and $N_0$ ... in other words the training data.

My question is this: since the posterior distribution is the prior probability times the likelihood of the training data, then is $\theta$ the vector of parameters that controls the prior probability? Also, how do we go about optimizing the first equation above to reach (or approach) being equivalent to the second equation?

I would also like to know what the difference is between the posterior predictive distribution and just the posterior.

Thanks in advance.

$\endgroup$
  • $\begingroup$ It is worth noting that there is a discrete distribution with finite support called the beta-binomial distribution, but it does not correspond to the conditional distribution of $\mathcal D \mid \theta$ that you wrote, which is simply binomial. Neither does it correspond to the prior distribution of $\theta$, which for your model is a conjugate prior if $\theta$ is beta distributed. In short, "beta-binomial" is different than saying "binomial likelihood with beta prior." $\endgroup$ – heropup Jul 13 '17 at 21:53
2
$\begingroup$

I don't know what you mean by "the vector of parameters that controls the prior probability". It appears that $\theta$ is simply a number between $0$ and $1$. The probability that $\theta$ is within any subset of the interval $(0,1)$ is the prior probability distribution; let us call it $Q(d\theta)$.

The posterior probability distribution is the conditional distribution given the number $N_1$ of heads and the number $N_0$ of tails. It is $$ c p(\mathcal D \mid \theta) Q(d\theta) = c \theta^{N_1} (1-\theta)^{N_0} Q(d\theta) $$ where $c$ is a normalizing constant given by $$ \frac 1 c = \int_0^1 \theta^{N_1}(1-\theta)^{N_0} Q(d\theta) \text{ so that } \int_0^1 c\theta^{N_1}(1-\theta)^{N_0} Q(d\theta) = 1. $$

To get $\widehat \theta_\text{MLE}$ you need to find the value of $\theta$ that maximizes $\theta^{N_1} (1-\theta)^{N_0}.$ Notice that $\theta^{N_1} (1-\theta)^{N_0}$ is $0$ if $\theta=0\text{ or }1$ and is positive if $0<\theta<1$ and is a continuous function of $\theta$ at $\theta=0$ and at $\theta=1$ and everywhere in between, and therefore it must have an absolute maximum in between. (Unless $N_1=0$ or $N_0=0,$ in which case it is not $0$ at one of the endpoints.) Since it is a differentiable function, the maximum other than at an endpoint must occur where the derivative is $0.$ If there is only one point where the derivative is $0,$ then that's it. We have $$ L(\theta) = \theta^{N_1}(1-\theta)^{N_0} $$ and $$ \ell(\theta) = \log L(\theta) = N_1\log\theta + N_0\log(1-\theta). $$ It is the usual practice to work on maximizing $\ell$ rather than $L$ because the derivative is simpler, and it yields the same solution because $\log$ is an increasing function. (And by $\log$ I mean the natural, or base-$e$, logarithm; you can write $\ln$ instead of $\log$ if you like.)

So we have $$\ell\,'(\theta) = \frac{N_1} \theta - \frac{N_0}{1-\theta}$$ and this equals $0$ only if $$ (1-\theta)N_1 = \theta N_0, $$ and thus $$ \theta = \frac{N_1}{ N_1+N_0} = \frac{N_1} N. $$

What is normally called the posterior distribution is the conditional distribution of $\theta$ given $N_1,N_0.$ The posterior predictive distribution gives the conditional probabilities of heads and tails on the next trial, given $N_1$ and $N_0.$ You have \begin{align} & \Pr(\text{heads on the next trial} \mid N_1,N_0) \\[10pt] = {} & \operatorname{E}( \Pr( \text{heads on the next trial} \mid \theta) \mid N_0, N_0) \\[10pt] = {} & \int_0^1 \Pr(\text{heads on the next trial} \mid \theta) c\theta^{N_1}(1-\theta)^{N_0} Q(d\theta) \\[10pt] = {} & \int_0^1 \theta \cdot c\theta^{N_1}(1-\theta)^{N_0} Q(d\theta) \end{align}

$\endgroup$
  • $\begingroup$ Sorry it took so long to reply ... I should have logged into the website over the weekend. Are you saying that since the derivative of the log likelihood is a decreasing function (asymptotes at zero), setting it equal to zero will show us what happens to theta as we maximize the log likelihood? $\endgroup$ – Josh Jul 17 '17 at 13:29
  • $\begingroup$ I'm not saying anything of the sort; I don't know how you got that. I explained why the likelihood function must have a maximum point somewhere in the interior of the interval from $0$ to $1.$ I said that since the logarithm function is increasing, the loglikelihood will have will have a maximum at that same point. And I said that if there is only one point at which the derivative is $0,$ then that is where the maximum is. $\endgroup$ – Michael Hardy Jul 17 '17 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.