Let $X_1,...,X_n$ be Bernoulli($p$) where $p$ is unknown, and $n>2$, and let $T=X_1+X_2$. My task is to calculate the information about $p$ in the entire sample and compare it to the information about $p$ given by the statistic.

After a few lines of work, I obtain the following expression for information contained in the sample: $$I_X(p)= n*E_p[(\frac{X_1p^{-1}(1-p)+X_1+1}{1-p})^2] $$

To calculate this, I first calculated the information given by one observation, and then multiplied that information by $n$.

Now, after some work, I obtained the following expression for information contained in $T$: $$I_T(p)=E_p[(\frac{Tp^{-1}(1-p)-1}{1-p})^2]$$

Now, I know from previous questions that $T$ is sufficient for $p$. Hence, $I_X(p)$ should equal $I_T(p)$. However, I have no idea how I am to compare these two quantities, because one has an $n$, and the other has an $X_2$ (in the $T$). Any advice?

  • $\begingroup$ Are you sure of the $I_X(p)$ expression? I have a hard time believing it's correct. $\endgroup$ – Ahmad Bazzi Sep 5 '18 at 19:09

Fisher Information Matrix (FIM), is the negative of the Expectation of the Hessian of the log likelihood function, namely \begin{equation} I(p) = - E H(p) \end{equation} where $I(p)$ is the FIM and $H(p)$ is the Hessian. Your likelihood function for independent samples is \begin{equation} L(p) = f(x_1\vert p) \ldots f(x_n \vert p) \end{equation} where \begin{equation} f(x_i \vert p) = p^{x_i} (1 - p)^{1 - x_i} \end{equation} Both equations above give us the likelihood function \begin{equation} L(p) = p^{ \sum x_i} (1 - p)^{n - \sum x_i} \end{equation} Log likelihood becomes \begin{equation} l(p) = \log L(p) = \log \big( p^{ \sum x_i} (1 - p)^{n - \sum x_i} \big) \end{equation} Denoting $\bar{x} = \frac{1}{n} \sum x_i$ for sake of simple presentation, one gets: \begin{equation} l(p) = n\bar{x} \log(p) + n(1 - \bar{x})\log(1-p) \end{equation} The score function (gradient of the log likelihood) is \begin{equation} s(p) = l'(p) = n \frac{\bar{x}}{p} - n \frac{(1- \bar{x})}{1 - p} \end{equation} The Hessian becomes \begin{equation} H(p) = s'(p) = - \frac{n(1-2p)\bar{x} + np^2}{p^2(1-p)^2} \end{equation} The FIM becomes \begin{equation} I(p)= -E H(p) = E\big[ \frac{n(1-2p)\bar{x} + np^2}{p^2(1-p)^2} \big] = \big[ \frac{n(1-2p)E\bar{x} + np^2}{p^2(1-p)^2} \big] \end{equation} But $E\bar{x} = p$ so \begin{equation} I(p) = E\big[ \frac{n(1-2p)\bar{x} + np^2}{p^2(1-p)^2} \big] = \big[ \frac{n(1-2p)p + np^2}{p^2(1-p)^2} \big] = \frac{n}{p(1-p)} \end{equation} Now, \begin{equation} T = X_1 + X_2 \end{equation} and the distribution of $T$ is \begin{equation} g(T=t\vert p) = Pr(T = t) = C_2^t p^t(1-p)^{n-t} \end{equation} Using Bayes theorem, we know that \begin{equation} f(x_1 \ldots x_n \vert T = t, p) = \frac{f(x_1 \ldots x_n,t=T \vert p)}{g(T= t \vert p)} \end{equation} Let's get $f(x_1 \ldots x_n,t=T \vert p)$, the likelihood function becomes \begin{equation} f(x_1 \ldots x_n , T = t\vert p) = p^{t + \sum\limits_{i=3}^n x_i}(1-p)^{n - t - \sum\limits_{i=3}^n x_i} \end{equation} \begin{equation} f(x_1 \ldots x_n \vert T = t, p) = \frac{f(x_1 \ldots x_n,t=T \vert p)}{g(T= t \vert p)} = \frac{p^{t + \sum\limits_{i=3}^n x_i}(1-p)^{n - t - \sum\limits_{i=3}^n x_i}}{C_2^t p^t(1-p)^{n-t}} \end{equation} which is \begin{equation} f(x_1 \ldots x_n \vert T = t, p) = \frac{p^{ \sum\limits_{i=3}^n x_i}(1-p)^{ - \sum\limits_{i=3}^n x_i}}{C_2^t} = \frac{1}{C_2^t} \Big(\frac{p}{1-p}\Big)^{\sum\limits_{i=3}^n x_i} = h(p) \end{equation} which is a function of $p$.

Hence, $T$ is NOT a sufficient statistics of $p$. However, if $T$ is the sum of ALL your samples, then you'd get a sufficient statistic.

  • $\begingroup$ Thanks for your response. What is a Hessian? $\endgroup$ – Roberto Sep 6 '18 at 11:56
  • $\begingroup$ Hi @DavidS .. it is the gradient of the gradient of a function. $\endgroup$ – Ahmad Bazzi Sep 6 '18 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.