Assume there are two confidence intervals for parameters $\theta_1$ and $\theta_2$, $C_1$ and $C_2$ with upper and lower bounds $[l_1,u_1]$ and $[l_2,u_2]$, respectively. Assume the intervals have nominal coverage $1-\alpha$.

The parameter estimates are $\hat{\theta}_1$ and $\hat{\theta}_2$. Now I always (in replications of the experiment) choose the interval with the higher estimate. Does this interval still have nominal coverage?

Formally define $$g:=\begin{cases} 1 \quad \text{if} \ \hat{\theta}_1 > \hat{\theta}_2 \\ 2 \quad \text{if} \ \hat{\theta}_1 \le \hat{\theta}_2 \end{cases}$$

with interval $C_g=[l_g,u_g]$. What is $P(\theta_d \in [l_g,u_g])$?

Remark 1: This seems to hold for two intervals, but fails for generalizations to more than two. I am looking for a proof of this result or a starting point.

Remark 2: I would also be interested if this problem is discussed anywhere in the literature and under which keyword / topic.

Remark 3: The problem can be understood in two ways. 1) The one population case where data $X_n$ are sampled from the same population and two different parameters are estimated on $X_n$ (e.g. mean and median, see BruceET's answer). 2) The two population case where $X_{n_1}$ and $X_{n_2}$ are sampled from two populations with $\theta_1$ and $\theta_2$ the same type of parameters (e.g. both means of potentially different values; this was my original intention with this question).

Edit: Based on BruceET's answer I programmed a simulation. It shows good nominal coverage of two-sided intervals. This simulation concerns the (1) scenario in remark 3.

true_mean = 100 # = true_median in case of normal distribution

cover_m = cover_med = cover_g = numeric()
for(i in 1:reps){
  x   = rnorm(n, true_mean, 10)
  m   = mean(x)
  med = median(x)
  g   = med > m
  CI_m   = t.test(x)$conf.int[1:2]
  CI_med = wilcox.test(x, conf.int=T)$conf.int[1:2]
  CI_g   = CI_m * (1-g) + CI_med * g
  cover_m[i]   = CI_m[1] <= true_mean & CI_m[2] >= true_mean
  cover_med[i] = CI_med[1] <= true_mean & CI_med[2] >= true_mean
  cover_g[i]   = CI_g[1] <= true_mean & CI_g[2] >= true_mean

apply( cbind(cover_m,cover_med,cover_g), 2,mean)

>cover_m cover_med   cover_g 
>0.95278   0.95097   0.95197
  • $\begingroup$ Suppose that the two types of CI, each taken separately, are equally valid both with coverage probability 95%. If you choose which type to use after seeing the data, then it is possible that your hybrid procedure has less than 95% coverage. // Exmp may be t CI for normal mean (=median) and Wilcoxon nonparametric CI for median. Criteria are different and may give different numerical results. Cherry-picking the one with most pleasing outcome may result is reduced coverage probability. $\endgroup$ – BruceET Apr 26 '18 at 6:05

This is not a proof of what I said in my Comment, but perhaps it gives some insight into what might go wrong, picking the most favorable estimate and its corresponding CI.

By trial and error using R statistical software, I found a sample of size $n=15$ from $\mathsf{Norm}(\mu=100, \sigma=10)$ for which a one-sided t CI incorrectly excludes $\mu = 100,$ while a one-sided Wilcoxon CI is correct. This is a case in which the sample mean is a little larger than the sample median, so (according to your suggested method) I would have chosen the incorrect t interval. [The same kind of error can happen with the more common two-sided confidence intervals, but it is easier to give an example with one-sided CIs.]

I believe choosing the larger of the two estimates will get non-convering CIs more often than sticking just with t CIs or just with Wilcoxon CIs.

x=rnorm(15, 100, 10); wilcox.test(x, conf.int=T, alte="g")$conf.int[1:2]; 
  t.test(x, alte="g")$conf.int[1:2]
[1] 99.6146     Inf      # Wilcoxon CI correct
[1] 100.4227    Inf      # t CI incorrect
mean(x); median(x)
[1] 103.9438             # mean larger than median ...
[1] 102.9209             #   ... so choose t CI
 [1]  93.94  94.40  94.55  96.36  97.15 101.69 102.08 102.92 104.06
[10] 109.13 109.39 109.96 111.00 115.79 116.74

I haven't seen this particular issue discussed in a published paper. Presumably a large-scale simulation could settle the issue for sure.

  • $\begingroup$ Thanks. I extended your code to a simulation, see edit above. It demonstrates good coverage for the case of two-sided intervals. A second point of thought is that you simulate data from a single population in which two parameters are defined. My initial idea was to have two populations with parameters $\theta_1$ and $\theta_2$ resepctively (e.g. both means of different sizes). I was not specific about this. Both problems (your interpretation and mine) are valid problems though. My simulation seems to contradict your post though? $\endgroup$ – tomka Apr 26 '18 at 7:50
  • 1
    $\begingroup$ Impressive simulation. (Already up-voted your interesting question once, so can't up-vote your simulation.) It shows that for two-sided CIs when choosing btw t and Wilcoxon your 'choose the biggest' strategy works OK. But that was the first comparison I thought of. Maybe not the best example. I still think there may be a risk with other pairs of CI types. Hope you can check out the particular case you are are using. $\endgroup$ – BruceET Apr 26 '18 at 8:06
  • $\begingroup$ Thanks -- I have tried it for a few different distributions for x (but always for means as parameters $\theta$ with the t-test). So far it works in all scenarios. I cannot prove this result though. Perhaps asymptotics play a role too, so that the result holds for normal sampling distributions. $\endgroup$ – tomka Apr 26 '18 at 8:56
  • $\begingroup$ With simulation you can sometimes show a particular situation works, but more general positive assertions require analytic proof. $\endgroup$ – BruceET Apr 26 '18 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.