Suppose I have $K$ independent test statistics $T_i$, with each derived from a sample of size $n_i$. It follows that $\sqrt{n_i} T_i\sim N(0,1)$ under the null.

If one were to use Bonferonni correction for multiple testing to control for family-wise error rate $\alpha$, each test would be allocated to a type I error rate of $\alpha/K$. Bonferonni is known to have a high type-II error. I understand that there are modified Bonferonni such as Holmes or Hochberg procedure. But they seem to be based on the rank of the p-values and may still be not very powerful.

Is there any known procedure (proposed somewhere) under such a scenario that allocates type I error based on relative sample size ${(n_i/n_k)}^{-1/2}$, when $n_i/n_k$ converges in some sense to a constant for any pairs of $i$ and $k$? That is we allocate $\alpha_i=\alpha \frac{n_i^{-1/2}}{\sum_{k=1}^K n_k^{-1/2}}$. How would this be compared to Bonferroni?

If the above is not a sensible procedure, why not?


Actually Bonferroni's remains true for any sequence $\alpha_1, \cdots, \alpha_K$ satisfying $\sum_{k=1}^{K}\alpha_k = 1$, which your formulation does satisfy. For example, if $p_1, \cdots, p_K$ were the $p$-values, then \begin{align*} \text{FWER} = P\left(\bigcup_{k=1}^{K}(p_k \le \alpha_k)\bigg|H_0\right) \le \sum_{k=1}^{K}P\left(p_k \le \alpha_k|H_0\right) = \sum_{k=1}^{K}\alpha_k = \alpha \end{align*} The more important question is if the choice of $\alpha_i = \alpha \frac{n_i^{-1/2}}{\sum_{k=1}^{K}n_k^{-1/2}}$ is optimal. Minimizing Type II error is equivalent to maximizing power. Our power under such a test would be \begin{align*} \beta &= P\left(\bigcup_{k=1}^{K}(p_k \le \alpha_k)\bigg|H_1\right) \\ &= 1 - P\left(\bigcap_{k=1}^{K}(p_k > \alpha_i)\bigg|H_1\right) \\ &\ge 1 - \bigwedge_{k=1}^{K}P(p_k > \alpha|H_1) \\ &= 1 - \bigwedge_{k=1}^{K}(1 - \beta_k) \\ &= \bigvee_{k=1}^{K}\beta_k \end{align*} where $\wedge$ and $\vee$ mean min and max, respectively. We can't exactly maximize the power, but we can at least maximize the lower bound. What this procedure implies is to simply allocate all of $\alpha$ to one of the $\alpha_k$ corresponding to the singular test with the most power. This is a bit underwhelming, but it guarantees a conversative control for the Type II error. A liberal control would be setting an upper bound for $\beta$, which can be derived from Bonferroni's inequality as \begin{align*} \beta \le \left(\sum_{k=1}^{K} \beta_k\right)\vee 1 \end{align*} So, we then optimize $\alpha_1, \cdots, \alpha_K$ to maximize the sum of the $\beta_k$'s.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.