What is a sufficient statistic?

I am trying to understand the definition of a sufficient statistic and trying to make conceptual sense of it. Wikipedia says $$Pr(X=x|T(X)=t,\theta) = Pr(X=x|T(X)=t)$$ Exactly how am I suppose to make sense of probability with $\theta$? Probability makes sense for $X$ because it is a function (a random variable) defined on a probability space. As far as I can tell $\theta$ has no sense of being defined on that probability space so how can a conditional make sense?

-

In statisitcal decision theory, you don't work with a single probability space. You have a measurable sample space $(\Omega,\Sigma)$ and a whole family of probability measures, $(\mu_\theta)$, indexed by $\theta$ on $(\Omega,\Sigma)$. In statistics, you don't know the underlying probabilities- you usually want to learn them.

Now the statistician has no direct information about $\theta$, but observes the distribution $\mu_\theta X^{-1}$ of a random variable $X$ that provides the data. Suppose now, that you have a statistic $T$, a measurable function of $X$. The statistic is sufficient if someone observing the distribution of $T\circ X$ learns no more about $\theta$ when observing the distribution of $X$ directly. So the distribution of $X$ conditional $T$, which you can calculate for each $\theta$ separately, doesn't show you anything about $\theta$, it is independent of $\theta$. There are different formulations of sufficiency, depending on different formulations of conditional probability.

The classic paper that formalized sufficient statistics within measure theoretic probability is Application of the Radon-Nikodym Theorem to the Theory of Sufficient Statistics by Halmos and Savage.

-
 I once heard it asserted by a moderately eminent mathematical statistician that there is a gap in the proof given by Halmos and Savage, consisting of the fact that they neglected to prove that the measure of certain sets is zero. He said the full proof was published independently by someone else. Bahadhur, maybe? – Michael Hardy Feb 23 '12 at 16:15 It's the first time I heard this. Bahadur might be a natural choice, I think he published the first follow-up paper to Halmos & Savage. – Michael Greinecker♦ Feb 23 '12 at 17:46 I'm not completely sure I understand what you are saying but would an equivalent way to phrase it be that there are random variables $X_\theta$ indexed by $\theta$ which are all functions on the same measurable space $(\Omega, \Sigma)$ but each with a different measure $\mu_\theta$ on it. And $X$ denotes $X_\theta$ for some particular $\theta$. Then $Pr(X_\theta=x|T(X_\theta)=t) = Pr(X=x|T(X)=t)$ where each side is computed with the same measurable space but with different measures. But is it necessary to introduce a whole bunch of $\mu_\theta$. Is it possible to just use one joint $\mu$? – user782220 Feb 23 '12 at 21:57 You can work with one probability measure and a family of random variables if all the $\mu_\theta$ are absolutely continuous with respect to a fixed measure. Roughly, that means when the all have a density. – Michael Greinecker♦ Feb 23 '12 at 22:59 So is the viewpoint $Pr(X_\theta=x|T(X_\theta)=t) = Pr(X=x|T(X)=t)$ then correct? – user782220 Feb 24 '12 at 1:36

It's not the probability of $\theta$ being in any particular set; it's a probability of some event given $\theta$.

Here's a simple example: $p\cdot100\%$ of all voters in a large population will vote "yes" in next week's referendum. A poll is based on a random sample of $n$ voters. Let $$X_i = \begin{cases} 1 & \text{if the }i\text{th voter will vote "yes"}, \\ 0 & \text{if not}. \end{cases}$$

Then the conditional probability distribution of $X_1,\ldots,X_n$ given $p$ and given the value of the sum $X_1+\cdots+X_n$ does not depend on $p$. Find the conditional probability and you'll find that $p$ cancels out. Therefore that sum is a sufficient statistic for $p$.

-

Perhaps a better way of thinking about it:

A statistic is sufficient for $\theta$ if the conditional distribution of X given T does not depend on $\theta$.

You're right in that $\theta$ is not a random variable.

-