Suppose I have $p$ null hypotheses $H_1,\ldots,H_p$ and the global hypothesis: $$ H_0:H_1 \cap H_2 \cap \cdots \cap H_p. $$ I'm interested in level controlling when testing for $H_0$ (using multiple testing): bounding $$ \Pr(\text{reject }H_0|H_0\text{ is true}) $$ regardless of the underlying dependence of the individual tests used to test $H_1,\ldots,H_p$. As this wiki article explains, I am in fact interested in controlling the family-wise error rate (FWER) in the weak sense.

The same article lists 2 methods of FWER level control (Bonferroni and Holm) that are robust to dependence of the individual tests. But these control FWER in the strong sense (defined in the article). So they work for my purpose but I'm afraid they are "overcompensating." What are some (modern) references for FWER level control in the weak sense?

  • $\begingroup$ How is defined as weak and strong? $\endgroup$ – V. Vancak Apr 30 '18 at 16:49
  • $\begingroup$ Weak = level control for the probability of rejecting a true null when all the nulls are true; strong = level control for the probability of rejecting a true null. $\endgroup$ – yurnero Apr 30 '18 at 16:51
  • $\begingroup$ @V.Vancak No. You should read the linked Wiki article and its subsidiary links. (This may save you time: in Wiki's notation, FWER deals with V whereas FDR deals with V/R.) $\endgroup$ – yurnero Apr 30 '18 at 16:53
  • $\begingroup$ The only modern approach that I aware of is variations of FDR. It does not fit to your task? $\endgroup$ – V. Vancak Apr 30 '18 at 16:56
  • $\begingroup$ @V.Vancak I don't think so, not for this specific task anyway. $\endgroup$ – yurnero Apr 30 '18 at 16:58

According to Remark 3.2 from [1], FDR coincides with weak FWER:

Given that all null hypotheses are null, any discovery is a false discovery, and therefore, $$FDP = V/R = 1.$$ ($V$ = # of false discoveries, $R$ = # of discoveries)

As you can see, under the condition that all nulls are true, the definitions of Type I error of a multiple test and FDP already coincide.

Since $FDR = E(FDP)$, when FDR is controlled at level $\alpha$, the Type I error given all nulls are true is controlled at level $\alpha$ as well.

Thus, FDR is a good choice for your problem.

[1] Rosenblatt, Jonathan: A Practitioner's Guide to Multiple Testing Error Rates. Available online at http://arxiv.org/pdf/1304.4920v3


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.