Intuitive Understanding of the Fisher Information? Within Statistics and Probability, the Fisher Information (https://en.wikipedia.org/wiki/Fisher_information) is generally said to describe the "amount of information that an observed random variable (i.e. a random data point) carries about some hidden parameter".
For example:

*

*If we assume a Uniform Probability Distribution with parameters (a,b) - any random point carries as much information as any other random point about the parameters (a,b).


*But in a Normal Probability Distribution with parameters (mu, sigma), points from the "tail" areas of the distribution are said to carry more information about the parameters (mu, sigma) compared to points closer to the "peak areas" (I am not exactly sure why this is).
Looking at the mathematical formula for the Fisher Information - it appears to be the "expected value of the square of the second derivative of the log likelihood function (for some probability distribution).
This leads me to my question: Why does the "expected value of the square of the second derivative of the log likelihood function (for some probability distribution)" tell us about the "information that an observed random variable carries about some hidden parameter"?
I can understand both of these arguments in isolation - I can accept that for some probability distributions, some random points are "more informative" about the probability distribution compared to other random points. And the mathematical formula used to calculate the Fisher Information seems (complex but) straightforward. But how does the "expected value of the square of the second derivative of the log likelihood function (for some probability distribution)" relate to amount of information carried by some random point?
Just by looking at the mathematical formula of the Fisher Information in isolation, how am I supposed to recognize that this is a formula that is describing the informativeness of a random point? It still seems a bit "arbitrary" to me, even though I am sure its not.
Thanks!
 A: I totally understand your confusion. First, let's make a distinction between Expected Fisher Information and Observed Fisher Information.
Expected Fisher Information, or just "Fisher Information" is a way to assess how useful observations of $X$ in $f(X;\theta)$ will be in locating the true value of $\theta$ -- it is the variance of the score function. At the true MLE, the expected value of the score function is 0 but it still has sampling variance. The closer the score function stays to 0, the more informative the variable $X$ is to determining the value of $\theta$.
The way we assess this "concentration" around the true value is to look at the curvature of the log-likelihood function at the MLE. This tells us how well our data identifies a particular range of $\theta$. High information implies low variance and high curvature, so we have a very peaked likelihood function around the MLE.
Why the second derivative? Because under very general conditions, the sampling distribution of the MLE will be normally distributed with mean $\theta$ and variance equal to the inverse of the observed fisher information. These asymptotic results are ultimately why we care about the second derivative of the log-likelihood.
In practice, we often use the Observed Fisher Information as it has better properties for inference, but the principle remains.
