1
$\begingroup$

Suppose that $X$ has a absolutely continuous and positive probability density function $f(x)$, I know its Fisher information is defined through $$I(X) := \int_{\mathbb R} \frac{|f'|^2}{f} \mathrm{d}x.$$ However, I am very confused about the definition of $I(X)$ when $X$ is a non-negative integer-valued random variable with probability mass function $(p_0,p_1,\ldots,p_n,\ldots)$. In most references I have found so far, it seems that the definition of the Fisher information contains some unknown parameter $\theta$ (for example, if $X$ is Bernoulli distributed, then the parameter is the success probability in each trial). Since I am not familiar with statistics, I am very confused as to how should we define Fisher information $I(X)$ when $X$ is a non-negative integer-valued random variable with (unknown) probability mass function $(p_0,p_1,\ldots,p_n,\ldots)$. Basically, my motivation is that I have a Fokker-Planck equation for the evolution of the probability mass function of a non-negative integer-valued stochastic process, and I want to know how its Fisher information evolves in time. Thanks for any help or reference!

$\endgroup$
2
  • $\begingroup$ Fisher information is a function of a parameter when a family of probability distributions is parametrized. $\endgroup$ Dec 23, 2020 at 19:18
  • $\begingroup$ For any probability density $f$, I can clearly define its Fisher information via $I(f) = \int_{\mathbb R} \frac{|f'|^2}{f} \mathrm{d}x$ (set to $\infty$ if this integral doesn't make sense), in this case, there is no "parameter" at all $\endgroup$
    – Fei Cao
    Dec 23, 2020 at 20:38

1 Answer 1

2
$\begingroup$

For a discrete known probability mass function, there is no parameter $\theta$—you know the full distribution. If however you know just the type or form distribution (such as a Gaussian, Bernoulli, etc.), you need to know the parameters (such as the sufficient statistics) in order calculate the Fisher Information (and other measures). After all, the Fisher Information (and the mean, and the variance, and...) of a Gaussian distribution depends upon the mean and the standard deviation, which in your terminology is $\theta$.

In the discrete case, every textbook on information theory will give the discrete version of the definition, in which an integral is replaced by a sum, for instance,

$$I[{\bf P}] = \sum\limits_{i=1}^n \frac{(p_{i} - p_{i-1})^2}{p_i}$$

as shown here.

$\endgroup$
2
  • $\begingroup$ Hi, thanks for your answer, but it is still vague to me. For a known probability density function $f(x)$ (let's say it's positive everywhere and absolutely continuous), I know its Fisher information is defined via $I(f) = \int_{\mathbb R} \frac{|f'|^2}{f} \mathrm{d}x$, there is no parameter as well. What would be the definition of Fisher information in the case of a general (unknown) discrete probability mass function $(p_0,p_1,\ldots,)$ then (with no parameter since it is a general pmf)? $\endgroup$
    – Fei Cao
    Dec 23, 2020 at 20:36
  • $\begingroup$ Yep.$\mbox{}\mbox{}\mbox{}$ $\endgroup$ Dec 28, 2020 at 6:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.