1
$\begingroup$

This is an extension to this previous question for this original question asked by myself:

When doing a maximum likelihood fit, we often take a ‘Gaussian approximation’. This problem works through the case of a measurement from a Poisson distribution. The Poisson distribution results in a likelihood for the average number $\mu$, given that $n$ events were observed:$L(\mu;n)=\cfrac{\mu^{n}e^{-\mu}}{n!}$.

We need to write this in the form $$L(\mu;n)\approx L_0 e^{-\frac{(\mu-n)^2}{2n}}$$ $\mu_0$ is defined to be the value of $\mu$ which gives the maximum of the likelihood, at which the likelihood has a value $L_0$.


Taking the natural logarithm of $L$ gives

$$\ln(L) = \ln(\mu^n)+\ln(e^{-\mu})-\ln(n!)=n\ln(\mu)-\mu - \ln(n!)$$

and taking the derivative gives

$$\frac{\mathrm{d}\ln(L)}{\mathrm{d}\mu}=\color{maroon}{\frac{n}{\mu}-1}$$

The maximum is given when $(2)$ is equal to zero; so $\mu_0$ satisfies $$\frac{n}{\mu_0}-1=0\implies \mu_0=n$$

Therefore $$\ln(L_0)=n\ln(n) - n -\ln(n!)$$

Performing a Taylor expansion up to the second order of $\ln(L)$ around the maximum, where $\mu_0=n$, is

$$\ln(L)\approx\ln(L_0) + \frac{\mathrm{d}\ln(L)}{\mathrm{d}\mu}\bigg|_{\mu=n}(\mu-n) + \frac{\mathrm{d^2}\ln(L)}{\mathrm{d}\mu^2}\bigg|_{\mu=n}\frac{(\mu-n)^2}{2!}$$

Taking the second derivative gives

$$\frac{\mathrm{d^2}\ln(L)}{\mathrm{d}\mu^2}=\color{#180}{-\frac{n}{\mu^2}}\tag{1}$$

Hence, to second order

$$\ln(L)\approx \ln(L_0) \color{blue}{- \frac{(\mu-n)^2}{2n}}$$

$$\implies \frac{L}{L_0}\approx e^{-\frac{(\mu-n)^2}{2n}}\implies \fbox{$\color{red}{L(\mu;n)\approx L_0 e^{-\frac{(\mu-n)^2}{2n}}}$}\tag{2}$$


The Gaussian approximation therefore comes down to neglecting the third order and higher terms. The first neglected term in the Taylor expansion is third order.

Evaluating this derivative gives $$\frac{\mathrm{d^3}\ln(L)}{\mathrm{d}\mu^3}=\color{#F80}{\frac{2n}{\mu^3}}$$ Hence, the first neglected term is $$\frac{\mathrm{d^3}\ln(L)}{\mathrm{d}\mu^3}\bigg|_{\mu=n}\frac{(\mu-n)^3}{3!}=\frac{2}{n^2}\frac{(\mu-n)^3}{6}=\frac{(\mu-n)^3}{3n^2}$$

For the approximation to be valid, this must be much smaller than the second order term, so

$$\fbox{$\color{purple}{\frac{(\mu-n)^3}{3n^2}\ll\frac{(\mu-n)^2}{2n}}$}\tag{???}$$

I have two simple questions regarding the expression marked $\color{purple}{\mathrm{purple}}$ since the second order term shown earlier in $\color{blue}{\mathrm{blue}}$ is negative: $\color{blue}{-\cfrac{(\mu-n)^2}{2n}}$. So the $\color{purple}{\mathrm{purple}}$ inequality cannot possibly be true.

However, assuming for my first question that we are only concerned with the magnitude (neglecting the negative sign); Why does the third order term need to be $\ll$ second order term? On what grounds is this necessary?

Thanks.

$\endgroup$
1
$\begingroup$

The first neglected term should be smaller in magnitude, so that $\big|\frac{(\mu-n)^3}{3n^2}\big| \ll \big|\frac{(\mu-n)^2}{2n}\big| $ or $|\mu-n| \ll \frac32 n $.

$\endgroup$
  • $\begingroup$ Excellent, that has answered my first question; Now can you please explain to me why the third order term needs to be $\ll$ second order term? Thank you. $\endgroup$ – BLAZE Dec 5 '15 at 10:21
  • $\begingroup$ That's so the first two terms are a good approximation to the actual sum. $\endgroup$ – marty cohen Dec 5 '15 at 22:23
1
$\begingroup$

Ok I'm gonna make a partial try, not sure at all that this is a fruitful approach, please feel free to point out anything fishy.

If a function is bounded on an interval or a set and has a Taylor expansion there, if we were not sure that new monomial terms had coefficients smaller than the previous at least such that $c_{k+1} \leq k xc_k$ for the $x$ in the area of convergence then the value could increase without bounds but that can not be because the function does not. Which $k$ would be sufficient for the series to diverge?

$\endgroup$
  • $\begingroup$ Thanks for your reply (+1), this was going to be a prime candidate for a 'tumbleweed' badge otherwise :) I'm glad that someone has taken the time to answer, I have to leave this open for a bit more input first though, thank you $\endgroup$ – BLAZE Dec 2 '15 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.