3
$\begingroup$

I have a very simple question about the following proof so you may not need to read all of this in order to answer the question which is right at the end; although I had to include the entire word for word copy of the question and solution to give the necessary context:

When doing a maximum likelihood fit, we often take a ‘Gaussian approximation’. This problem works through the case of a measurement from a Binomial distribution: $$L(n;p,N)=\cfrac{N!}{n!(N-n)!}p^n(1-p)^{N-n}$$ $p_0$ is defined to be the value of $p$ which gives the maximum of the likelihood, at which the likelihood has a value $L_0$.

$\fbox{$\color{purple}{\text{The objective is to find the value of the standard deviation $\sigma$ for this Gaussian approximation:}}$}$ The solution goes as follows:

Taking natural logs gives $$\ln(L) = n\ln(p) + (N-n)\ln(1-p)- \ln\left(\frac{N!}{n!(N-n)!}\right)$$ The first derivative is $$\frac{\mathrm{d}\ln(L)}{\mathrm{d}p}=\frac{n}{p}-\frac{N-n}{1-p}$$ The maximum $p_0$ is given when $\cfrac{\mathrm{d}\ln(L)}{\mathrm{d}p}=0$ $$\implies \frac{n}{p_0}=\frac{N-n}{1-p_0}\implies n-np_0=p_0N-p_0n\implies \fbox{$\color{blue}{p_0=\frac{n}{N}}$}\tag{1}$$ Taking the second derivative gives $$\frac{\mathrm{d^2}\ln(L)}{\mathrm{d}p^2}=-\frac{n}{p^2}-\frac{N-n}{(1-p)^2}$$ The Taylor expansion to second order is $$\ln(L)\approx\ln(L_0) + \frac{\mathrm{d}\ln(L)}{\mathrm{d}p}\bigg|_{p=\color{blue}{p_0}}\times\frac{\left(p-\color{blue}{p_0}\right)}{1!} + \frac{\mathrm{d^2}\ln(L)}{\mathrm{d}p^2}\bigg|_{p=\color{blue}{p_0}}\times\frac{\left(p-\color{blue}{p_0}\right)^2}{2!}$$ $$\implies\ln(L)\approx\ln(L_0) + {\left(\frac{n}{\color{blue}{p_0}}-\frac{N-n}{1-\color{blue}{p_0}}\right)}\left(p-\color{blue}{p_0}\right)-{\left(\frac{n}{\color{blue}{p_0}^2}+\frac{N-n}{(1-\color{blue}{p_0})^2}\right)}\frac{(p-\color{blue}{p_0})^2}{2}$$ $$\require{enclose}\implies\ln(L)\approx\ln(L_0) + \enclose{updiagonalstrike}{\underbrace{\left(\frac{n}{\color{blue}{\frac{n}{N}}}-\frac{N-n}{1-\color{blue}{\frac{n}{N}}}\right)}_{=0}\left(p-\color{blue}{\frac{n}{N}}\right)}-\frac12{\left(\frac{n}{\left(\color{blue}{\frac{n}{N}}\right)^2}+\frac{N-n}{\left(1-\color{blue}{\frac{n}{N}}\right)^2}\right)}\left(p-\color{blue}{\frac{n}{N}}\right)^2$$ $$\implies\ln(L)\approx\ln(L_0) -\frac12{\left(\frac{N^2}{n}+\frac{N^2}{N-n}\right)}\left(p-\frac{n}{N}\right)^2$$ $$\implies\ln(L)\approx\ln(L_0) -\left(\frac{N^3\left(p-\frac{n}{N}\right)^2}{2n(N-n)}\right)$$ $$\implies\ln\left(\frac{L}{L_0}\right)\approx -\left(\frac{N^3\left(p-\frac{n}{N}\right)^2}{2n(N-n)}\right)$$ $$\implies L\approx L_0\exp{\left(-\frac{\left(p-\frac{n}{N}\right)^2}{\left(2\times\frac{n(N-n)}{N^3}\right)}\right)}$$ So $$\sigma=\sqrt{\frac{n(N-n)}{N^3}}$$ This does not at first sight correspond to the standard deviation obtained from the binomial distribution. This is because that is in terms of $n$, whereas the result derived above is in terms of $p$. $\color{red}{\text{However, since in the approximation used here}}$ $\color{red}{p\approx\frac{n}{N}}$ $\color{red}{\mathrm{,}}$ $\color{red}{\text{then the standard deviation in terms of $n$ will be $N$ times the above, i.e}}$ $$\color{red}{\sigma =N\sqrt{\frac{n(N-n)}{N^3}}} =\sqrt{\frac{n(N-n)}{N}}=\sqrt{N\frac{n}{N}\left(1-\frac{n}{N}\right)}=\sqrt{Np\left(1-p\right)}$$ as required.

I have completely understood all of the above proof apart from the part marked $\color{red}{\mathrm{red}}$.

First of all; $p_0=\frac{n}{N}$ as given in $(1)$ so I would like to know the justification as to why $\color{red}{p\approx\frac{n}{N}}$?

Secondly, what on earth is the author talking about when they say that the standard deviation

"$\color{red}{\text{will be $N$ times the above}}$"?

Since when is multiplying $\sigma$ by $N$ justified?

$\endgroup$
2
+50
$\begingroup$
  1. It was not given that $p_0 = \frac{n}{N}$, it was shown using the maximum likelihood method. That is the justification. Since this is MLE, it makes sense that $p\approx p_0$.

  2. It looks like he is saying $$\sigma^2 =\text{Var}(p) = \text{Var}(p_0) = \text{Var}\left(\frac{n}{N}\right) = \frac{1}{N^2}\text{Var}(n).$$ This implies $\text{Var}(n) = N^2\text{Var}(p)$, and finally $$\text{SD}(n) =N\times\text{SD}(p) = N \sigma. $$

$\endgroup$
  • $\begingroup$ Thanks for your response; what is MLE? $\endgroup$ – BLAZE Dec 14 '15 at 9:26
  • 1
    $\begingroup$ No problem. It means maximum likelihood estimator. $\endgroup$ – Em. Dec 14 '15 at 9:27
  • $\begingroup$ I have understood everything you wrote; apart from why $\text{Var}\left(\frac{n}{N}\right) = \frac{1}{N^2}\text{Var}(n)$? Could you please elaborate on this a little? Thanks. $\endgroup$ – BLAZE Dec 21 '15 at 7:48
  • 1
    $\begingroup$ "Apart from why $\text{Var}\left(\frac{n}{N}\right) = \frac{1}{N^2}\text{Var}(n)$?"?? I'm not sure what you're asking. I think you should be aware that if there is some random variable $X$, and some constant $c$, then $\text{Var}(cX) = c^2 \text{Var}(X)$. In this case $N$ is a constant. You always take a sample of size $N$. However, $n$ denotes the number of successes, and thus it is random. The number of success is random/different every time you take a sample of size of size $N$. Hence, in this case, $n$ is like a random variable $X$. $\endgroup$ – Em. Dec 21 '15 at 7:55
  • 1
    $\begingroup$ No, it doesn't. Have a look at item 6 on this page for a proof. $\endgroup$ – Em. Dec 21 '15 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.