5
$\begingroup$

Let $X_1,...,X_n$ be Bernoulli distributed with unknown parameter $p$.

My objective is to calculate the information contained in the first observation of the sample.

I know that the pdf of $X$ is given by $$f(x\mid p)=p^x(1-p)^{1-x}$$, and my book defines the Fisher information about $p$ as

$$I_X(p)=E_p\left[\left(\frac{d}{dp}\log\left(p^x(1-p)^{1-x}\right)\right)^2\right]$$

After some calculations, I arrive at

$$I_X(p)=E_p\left[\frac{x^2}{p^2}\right]-2E_p\left[\frac{x(1-x)}{p(1-p)}\right]+E_p\left[\frac{(1-x)^2}{(1-p)^2}\right]$$

I know that the Fisher information about $p$ of a Bernoulli RV is $\frac{1}{p(1-p)}$, but I don't know how to get rid of the X-values, since I'm calculating an expectation with respect to $p$, not $X$. Any clues?

$\endgroup$
  • 1
    $\begingroup$ $+1$ for showing your work to derive the correct $I_X(p)$ $\endgroup$ – Ahmad Bazzi Sep 16 '18 at 15:56
5
$\begingroup$

\begin{equation} I_X(p)=E_p \left[\frac{X^2}{p^2}\right]-2E_p \left[ \frac{X - X^2}{p(1-p)} \right] + E_p \left[ \frac{X^2 - 2X + 1}{(1-p)^2}\right] \tag{1}. \end{equation} For a Bernoulli RV, we know \begin{align} E(X) &= 0(\Pr(X = 0)) + 1(\Pr(X = 1)) = p\\ E(X^2) &= 0^2(\Pr(X = 0)) + 1^2(\Pr(X = 1)) = p. \end{align} Now, replace in $(1)$, we get \begin{equation} I_X(p)=\frac{p}{p^2}-2\frac{0-0}{p(1-p)}+\frac{p-2p+1}{(1-p)^2} = \frac{1}{p}-\frac{p-1}{(1-p)^2} = \frac{1}{p} - \frac{1}{p-1} = \frac{1}{p(p-1)}. \end{equation}

$\endgroup$
  • $\begingroup$ thanks @MichaelHardy $\endgroup$ – Ahmad Bazzi Sep 16 '18 at 18:00
  • $\begingroup$ I always thought that when an expectation is written as $E_p$, for example, it means we treat the other variables (not p) as constants. Apparently not? $\endgroup$ – Roberto Sep 16 '18 at 18:23
  • 1
    $\begingroup$ Yes i understand what you mean. I usually do not write $E_p()$. I write $E_{X \vert p}()$, where the expectation is taken with respect to the samples given the parameters you want to estimate, i.e. $\endgroup$ – Ahmad Bazzi Sep 16 '18 at 18:24
  • 1
    $\begingroup$ The expectation is taken with respect to any random variable (whether is $X$, $Y$ or whatever other r.v. $E_p$ only intends to remark the fact that your model for the distribution of $X$ is not fully specified, but it is uncertain up to a parameter $p$, and thus the corresponding expectations taken may depend on that value $p$. $\endgroup$ – Alejandro Nasif Salum Sep 16 '18 at 18:38
  • 1
    $\begingroup$ Thanks @AlejandroNasifSalum for the edit $\endgroup$ – Ahmad Bazzi Sep 17 '18 at 23:28
2
$\begingroup$

Actually, the Fisher information of $X$ about $p$ is $$I_X(p)=E_p\left[\left(\frac{d}{dp}\log f(X\mid p) \right)^2 \right],$$ that is $$I_X(p)=E_p\left[\left(\frac{d}{dp}\log\left(p^X(1-p)^{1-X}\right)\right)^2\right].$$

I've only changed every $x$ by $X$, which may seem as a subtlety, but then you get $$I_X(p)=E_p\left(\frac{X^2}{p^2}\right)-2E_p\left(\frac{X(1-X)}{p(1-p)}\right)+E_p\left(\frac{(1-X)^2}{(1-p)^2}\right).$$

The expectation is there for the fact that $X$ is a random variable. So, for instance: $$E_p\left(\frac{X^2}{p^2}\right)=\frac{E_p\left(X^2\right)}{p^2}=\frac{p}{p^2}=\frac1p.$$

Here I used the fact that $E_p(X^2)=p$, which can easily be seen as $$E_p(X^2)=0^2\cdot p_X(0)+1^2\cdot p_X(1)=0^2(1-p)+1^2p=p,$$ or by the observation that $X\sim \operatorname{Be}(p) \implies X^n\sim \operatorname{Be}(p)$ as well. Then you can go on with the remaining terms.


Additionally, an equivalent formula can be proved for $I_X(p)$ given the second derivative of $\log f$ is well defined. This is $$I_X(p)=-E_p\left(\frac{d^2}{dp^2}\log f(X\mid p) \right),$$ and many times you'll get simpler expressions. In this case, for instance, you get $$I_X(p)=-E_p\left(\frac{d^2}{dp^2}\log p^X(1-p)^{1-X}\right)=$$ $$=-E_p\left(-\frac X{p^2}-\frac{1-X}{(1-p)^2} \right) = \frac {E_p(X)}{p^2}+\frac{E_p(1-X)}{(1-p)^2}=$$ $$=\frac {p}{p^2}+\frac{1-p}{(1-p)^2}=\frac 1p+\frac 1{1-p}=\frac 1{p(1-p)},$$ as desired.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.