Interpretation of sampling distribution from Frequentist point of view

Question

I've read multiple of the threads about Bayesian vs Frequentist interpretations of probability, but I'm having trouble trying to reconcile them with the idea of the sampling distribution when performing parameter estimation. Here is what is going on in my head:

Say we want to fit a probability distribution to a collection of data. Examining the data we choose a probability distribution (say the Poisson as an example). To fit the distribution to our data we have to obtain an estimate for the population parameter $\lambda$. So we obtain the estimate using one of the known procedures (e.g. Maximum Liklihood Estimation). We obtain an estimate for the true parameter value, $\hat{\lambda}$. Since our estimate $\hat{\lambda}$ is a function of our sampled data this implies it has a probability distribution, which is commonly referred to as the sampling distribution.

Now this is where I'm having trouble tying things together. My interpretation of what a sampling distribution was that it "represents the possible values that the true parameter value could take on with respective probability attached to them." But also from the frequentist approach the parameter values are constant and as such have no probability attached to them.

But we obtained these parameter estimates from a random process of sampling, so how can they not have probability attached to them?

As can be seen there a few things I'm confused about. Some clarification would be much appreciated.

Answer 1

In the frequentist interpretation, parameters are fixed but unknown, so it is correct to say that they are not stochastic. We infer their values from realizations of the distribution they model, and this is what we call parameter estimation. The fundamental question is, "where is the randomness?"

A frequentist answers this question by saying the parameters are fixed but the observations/data are realizations of some underlying random process. The data are random in the sense that they change from experiment to experiment by virtue of some random sampling process. Therefore, any statistic we calculate from the data inherits that randomness, including any statistics that are parameter estimates. This includes point estimates, interval estimates, and so forth.

So the sampling distribution of an estimator, or of any statistic for that matter, is the probability distribution of that statistic. It is not, as you have put it, a representation of "the possible values that the true parameter value could take on," in as much as it is incorrect to speak of a confidence interval's coverage probability as the probability that the true value of the parameter falls within a given calculated interval.

Sampling distributions are distributions on statistics, and in the case where a statistic is an estimator of a parameter, they are distributions on those estimators. They are not distributions of parameters.

To illustrate, it is a rather unfortunate and common abuse of language to use the phrase "sampling distribution of the mean" when in fact the terminology ought to be "sampling distribution of the sample mean." The former is misleading because it suggests that a distribution exists for the parameter(s) or some function thereof (the mean of the distribution), rather than for a statistic obtained from data, which is the sample mean. It seems redundant to use "sampling"/"sample" but I argue that the redundancy here is "sampling," not "sample," since the meaning of "distribution of the sample mean" cannot be misunderstood as anything else than a sampling distribution. Similarly, we can construct sampling distributions for the standard error (of the mean), or for the sample median, and it is not at all ambiguous to say "distribution of the sample median" without prefacing it with "sampling."

When we use a sampling distribution for constructing an interval estimate, e.g., the standard error of the sampling distribution of the sample mean may give us a confidence interval for a parameter, the resulting estimate is itself a random variable (or pair of random variables, if you want to think of it that way). The coverage probability is the probability that the interval contains the true value of the parameter. The distinction is subtle but important: here, the parameter is fixed, and it is the interval that varies randomly from one experiment/sample to another, and the coverage probability is what proportion of such samples on average will produce an interval estimate which contains the parameter being estimated.