I'm a little baffled by p-values; they seem to be very informally defined in the textbooks, websites, and other resources to which I've been exposed. For example, the common definition:

"The p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true."

... is very inadequate. What does "extreme" mean, and how can we tell if some theoretical observations are more "extreme" than others? If we're working with random variables that follow a vaguely bell-shaped pdf or pmf, then we can simply define p-values as Wikipedia does:

$P(X \leq x \mid H_{0})$ for left-tailed events,

$P(X \geq x \mid H_{0})$ for right-tailed events, and

$2 \min \{ P(X \leq x \mid H_{0}), P(X \geq x \mid H_{0}) \}$ for two-tailed events.

But we don't always work with nice, pretty random variables that follow nice, pretty bell-shaped distributions. Let's say I have a die that I purchase from a prank shop. The die was advertised as only rolling 6s and 1s, each 50% of the time. However, I then hear that the manufacturer of my die made a mistake; some of their weighted dice solely roll 6s. I decide to test to see if the die is among those. The null hypothesis is that $P(1) = P(6) = \frac{1 - 4a}{2}$, where $a$ is some very small number indicating the chance of rolling a 2, 3, 4, or 5 individually. The alternative hypothesis is $P(6) = 1 - 5a$, where the probability of all non-sixes is $a$.

In this scenario, rolling a whole bunch of 6s looks very bad for the null hypothesis. But so does rolling a truckload of 2s, 3s, 4s, and 5s! Evidence that the die is weighted fairly, despite not supporting the alternative hypothesis, does not and should not factor into the p-value. It's difficult, if not impossible, to talk about the "extremeness" of certain results over others because results that reflect badly on the null hypothesis aren't numerically higher or lower than ones that do not. Therefore, while I'm tempted to claim that the p-value in this scenario should be calculated using a two-tailed test rather than a one-tailed test, even the two-tailed test doesn't seem applicable.

I've given this some thought, and I've concluded that it is possible to construct a formal definition for p-values resulting from a "two-tailed" test (or rather, a test that collapses to a two-tailed test when potential results follow a symmetric, vaguely bell-shaped distribution), but I have no idea how to go about constructing a formal definition for p-values from a one-tailed test. For the "two-tailed" version, I came up with:

$p = \sum_{w \in B} P(w \mid H_{0})$ such that $B = \{ u \in \Omega \mid P(u \mid H_{0}) \leq P(x \mid H_{0}) \}$

where $p$ is the p-value, $\Omega$ is the set of all possible results, and $x$ is the actual result found by your test.

As I said earlier, this definition of a p-value is identical to the two-tailed version when results follow symmetric, vaguely bell-shaped distributions. It works for discrete and continuous variables alike, if you're willing to be non-uppity about your definition of an integral ("the sum of all values between two bounds, multiplied by an infinitesimal" may not be very rigorous, but it works for our purposes and allows us to treat continuous variables like discrete ones). However, I cannot think of a way to formalize the definition of p-values for one-tailed tests... and I have a sneaking suspicion that it's impossible to do so.

Typically, when textbooks/articles talk about when to use one-tailed tests vs two-tailed tests, they do so with the alternative hypothesis in mind. Here's a common example: we suspect a man has psychic powers, and is able to tell the contents of boxes without looking inside them. We decide to test him. We give the man two boxes, one filled with a prize, and one that is empty. We then tell him to choose the box filled with the prize.

If the null hypothesis is true, then he'll only find the prize 50% of the time. However, we'd assume that if the alternative hypothesis were true, then the man will find the prize more than 50% of the time. What if he finds it less than 50% of the time?

A common textbook response is to say that we should use a one-tailed test--after all, the idea that he's psychic enough to know which box contains the prize but not psychic enough to pick the right box seems kind of silly. However, what if our alternative hypothesis were that the man is watched over by a powerful deity, who grants him extreme luck or curses him with very poor luck depending on the day (strange hypothesis, I know, but bear with me). Unusually high and low rates of selecting the correct box are both consistent with the alternative hypothesis, because we don't know what kind of luck the deity has decided to grant the man on testing day. The textbooks would likely argue that a two-tailed test is in order here.

Perhaps I'm wrong, but this seems completely orthogonal to the purpose of p-values. They're supposed to minimize the chance of rejecting a correct null hypothesis. My choice of $H_{A}$ should have no effect on this goal, because only the null-hypothesis is "on trial." Therefore, I've concluded that one-tailed tests cannot be formally defined, because the choice of "tail" is entirely dependent on the choice of alternative hypothesis, which should play no role in calculating the p-value.

My question is two-fold. 1) Is my proposed definition for "two-tailed" tests acceptable, and is this what statisticians typically use when confronted with non-bell shaped distributions (if my method isn't what statisticians use, what DO they use?), and 2) is there a way to construct a formal definition of a p-value from a one-tailed test, and if so, what is it?

  • $\begingroup$ The P-value is the probability under $H_0$ of a result as extreme or more extreme than the observed result, where 'extreme' means in the direction(s) of the alternative. It is not true that "the purpose of a P-value is to minimize the chance of rejecting a true null hypothesis." The P-value is a statement about the data-- to be used for inductive statistical reasoning, not a statement about the probability that the null hypothesis is true. You may not like or understand the definition of P-value, but your 'discussion' of it is not useful for statistical inference. $\endgroup$ – BruceET Apr 10 '17 at 7:50
  • $\begingroup$ @BruceET I'm sorry, I'm trying to learn how p-values work as best as I can. =( If possible, I'd like to ask some clarifying questions: Does "in the direction of the alternative" mean that P(x | H_A) > P(x | H_0)? I'm looking for more formal characterizations of results than whether they are in the "direction" of one hypothesis or another. And Wikpedia states that "When the p-value is calculated correctly, this test[rejecting null when p is below α] guarantees that the Type I error rate is at most α." You seem to disagree with this statement; am I misreading Wikipedia, or is it incorrect? $\endgroup$ – SilasLock Apr 12 '17 at 3:18
  • $\begingroup$ Examples: (a) $H_0: \mu = 0$ vs $H_a: \mu > 0:$ Reject for large values of $\bar X_{obs}$; P-value is probability (under $H_0$) of getting an $\bar X \ge \bar X_{obs}$. (b) $H_0: \mu = 0$ vs $H_a: \mu < 0:$ Reject for small (neg) values of $\bar X_{obs}$; P-value is probability (under $H_0$) of getting an $\bar X \le \bar X_{obs}$. (c) $H_0: \mu = 0$ vs $H_a: \mu \ne 0:$ Reject for large values of $|\bar X_{obs}|$; P-value is probability (under $H_0$) of getting an $|\bar X| \ge |\bar X_{obs}|$. That is sum of probabilities in two tails.// No disagreement with Wikipedia; different issue. $\endgroup$ – BruceET Apr 12 '17 at 6:29

There is an approach to defining a $p$-value that doesn't care about the existence or type of an alternative hypothesis, the number of tails your tests have, and just about anything else about the problem. It is simply this:

A $p$-value is a test statistic with values between $0$ and $1$ which, assuming the null hypothesis, has the uniform distribution.

For anything that satisfies this definition, it is true that $\Pr[ p < 0.05 \mid H_0] = 0.05$. As a result, if you follow the policy "reject the null hypothesis whenever the $p$-value is less than $0.05$", you will reject the null hypothesis in about $5\%$ of the cases where the null hypothesis is actually correct. (This is the thing called Type I errors, if like me you have trouble keeping track which type is I and which type is II.)

You may have noticed some things... missing from this definition. For example, under this assumption, ignoring the data and picking a random real number from $[0,1]$ produces a $p$-value.

We can say that this is a valid $p$-value for a test with very low statistical power, where statistical power is defined as our probability of rejecting the null when the alternative hypothesis is true. This is the step that cares about the alternative hypothesis and the possibility of false negatives (Type II errors). It's also practically impossible to calculate statistical power without making lots of assumptions.

So both the one-tailed and the two-tailed $p$-value are always valid, but depending on your assumption about the alternative hypothesis, one of them may have higher statistical power. As long as we don't draw any conclusions from the $p$-value that don't follow from the "has a uniform distribution" definition, we're safe.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.