Example of Sufficient and Insufficient Statistic?

I am having trouble understanding the concept of a sufficient statistic. I have read What is a sufficient statistic? and Sufficient Statistic (Wikipedia)

Can someone please give an example of:

1. a simple (but non-trivial) statistical model
2. a sufficient statistic of that model
3. an insufficient statistic of that model
4. how you identified 2 & 3 as having and lacking, respectively, the sufficiency property
-

$\def\E{\mathrm{E}}$Consider samples $X = (X_1,X_2)$ from a normally distributed population $N(\mu,1)$ with unknown mean.

Then the statistic $T(X)=X_1$ is an unbiased estimator of the mean, since $\E(X_1)=\mu$. However, it is not a sufficient statistic - there is additional information in the sample that we could use to determine the mean.

How can we tell that $T$ is insufficient for $\mu$? By going to the definition. We know that $T$ is sufficient for a parameter iff, given the value of the statistic, the probability of a given value of $X$ is independent of the parameter, i.e. if

$$P(X=x|T=t,\mu)=P(X=x|T=t)$$

But we can compute this:

$$P(X=(x_1,x_2) | X_1=t,\mu) = \begin{cases} 0 & \mbox{if }t\neq x_1 \\ \tfrac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(x_2-\mu)^2} & \mbox{if }t=x_1 \end{cases}$$

which is certainly not independent of $\mu$.

On the other hand, consider $T'(X) = X_1+X_2$. Then we have

$$P(X=(x_1,x_2) | X_1+X_2=t, \mu) = \frac{1}{2\pi}\int_{-\infty}^{\infty}e^{-\frac{1}{2}(s-\mu)^2 - \frac{1}{2}(t-s-\mu)^2}ds$$

and you can complete the square and show that this is independent of $\mu$, and hence $T'$ is a sufficient statistic for the mean $\mu$.

-
Part way through writing this, I wished I'd chosen a Bernoulli distribution with parameter $p$ instead of a normal distribution. It would have been easier to understand, less effort for me to type, and I would have had the patience to do the integral at the end myself. Let me know if you still don't understand, and I'll rewrite my answer. –  Chris Taylor May 17 '12 at 11:33
I think it's clicked - the statistic is a function of the sample. If given the statistic result on the sample, the unknown model parameter becomes conditionally independent of the sample, the statistic is sufficient. –  Andrew Tomazos May 17 '12 at 11:43
Almost - if, given the statistic, the sample is independent of the parameter, then the statistic is sufficient. Conditional independence and independence are different things. –  Chris Taylor May 17 '12 at 11:45
How are they different in this context? I thought the conditioning is on the knowledge of the statistic (this is the third event). –  Andrew Tomazos May 17 '12 at 13:32
The RHS of the last displayed equation should read $\exp(-(x_1-t/2)^2)/\sqrt{\pi}$ if $x_1+x_2=t$ and $0$ otherwise (thus the argument is correct although the formula in the RHS is not). –  Did May 20 '12 at 9:36

Just look at the form of an exponential family of distribution. The function of the data in the exponent is the sufficient statistic. The most obvious case is the sample mean for a normally distributed data with a known variance. The sample mean is a sufficient statistic for the population mean. The full information in the data is the n observations X1, X2,...,Xn but there is no additional information in that data that will help in the estimation of the population mean given the sample mean. when the variance is unknown the sample mean and the sample variance represent the sufficient statistic for the population mean and variance. Sufficiency is important because it plays a major role in the theory of parametric point estimation. An insufficient statistic would be any statistic different from the sufficient one. So in the normal distribution for example let Y1=X1=X2, Y2=X3-X4,... Ym=Xn-1 -Xn for m=n/2 (where say n is even). Then Y1, Y2,...,Ym is not sufficient for the mean and variance of the normal. So this answers 1-3.

-
What is item (3) in your answer? –  Andrew Tomazos May 17 '12 at 11:13
Is that a typo, you mean "Y1=X1-X2" ? How did you determine that your example for 3 was not sufficient? –  Andrew Tomazos May 17 '12 at 11:24
and also is the last element meant to be "Ym = Xm-1 - Xm" ? –  Andrew Tomazos May 17 '12 at 11:30
I have not answered part 4. I think that takes a longer explanation and I haven't found the time to do that yet. The easy and unsatisfying answer though is that it does not contain the sufficient statistic X bar. I meant Ym=Xn-1-Xn I used the notationm just to avoid n/2 as a subscript. –  Michael Chernick May 17 '12 at 13:55

Here is my attempt to answer part 4 of the question. If you have a parameteric family of distributions with a parameter theta (theta can be K-dimensional for K>=1) then a statistic is sufficient if all the information about theta is contained in it. That means that given the parametric family the conditional distribution of the data given the sufficient statistic is independent of the parameter theta. For many problem there are several functions T of the data that are sufficient. Sufficiency usually carries data reduction with it. For example say the parametric family is N(m,1) (m is the mean and 1 is the variance). To estimate m you take a sample of size n=10. The data involves ten values but the sufficient statistic (the sample mean) is just a single number. That is the least amount of information possible for a sufficient statistic. This type of sufficient statistc is better than others in data reduction and is called minimal sufficient. To determine that a candidate statistic is sufficient you can use the factorization theorem to test it. If the candidate T is sufficent it will factor the density as follows f(x|theta)= g(T(x)|theta) h(x) where f is the density for the data given theta, g is the density for T(x) given theta and the term h(x) does not depend on theta. So you can check sufficiency by calculating f and g, divide f by g and T will be sufficient iff the resulting function h doesn't depend on (involve) theta. Besides sufficiency and minimal sufficiency there is a concept called complete sufficiency. All the cocepts are important in the parametric theory of point estimation. This is nicely covered in Casella and Berger's book Statistical Inference 2e Chapter 6.

-