# What are useful tests of differences between non-normal samples.

Sample 1 $X_i$ has $n_1$ observations from Population 1 and, independently, Sample 2 $Y_i$ has $n_2$ observations from Population 2. What some reasonable tests that the two populations differ: perhaps as to their medians $\eta_1$ and $\eta_2,$ perhaps as to their means $\mu_1$ and $\mu_2,$ or perhaps in some other specified manner.

Background: A recent question about testing supposedly non-normal data seemed unclearly stated. My idea was that OP was asking for a two-sample nonparametric test, and I answered accordingly. After discussion comments, it seemed that no two-sample test was sought. The question was put on hold and I am deleting my answer. My Answer below is a somewhat generalized version of the deleted answer. I could find no suitably-labeled discussion of nonparametric 2-sample tests on our site.

• Are the distribution on discrete sets? – Clement C. Mar 2 '18 at 19:38
• Not necessarily. The Wilcoxon test assumes continuous data (as noted), but the permutation test is more general. If you have a suggestion for a test limited to discrete populations, please consider posting it. – BruceET Mar 2 '18 at 19:44

First, one should verify that there are two independent samples $X_i$ and $Y_i$ from two populations, not pairs of data $(X_i, Y_i)$ from one bivariate distribution.

Clues to detect paired data: (a) Paired data typically show association between the $X$s and the $Y$s. Perhaps make a scatterplot of $Y_i$ against $X_i$ and look to see if there is a pattern of association, or compute the sample correlation $r_{X,Y}$ to see if it is significantly different from $0$. (b) Either paired or independent $X_i$ and $Y_i$ may have equal sample sizes $n.$ But unless there are missing observations, a paired design must have equal samples sizes.

Second, do not automatically assume that the populations from which the data were sampled are too far from normally distributed to use a two-sample t test. Although the theory behind t tests is based on the assumption of normal populations, slight departures from normality may not spoil results from a t test. One says that t tests are 'robust' against mild non-normality. If sample sizes are small (below about 20), it may not be feasible to do formal tests of normality. Obvious skewness or several far outliers in the same tail are some contraindications for use of t tests. For larger samples, one might make normal probability plots or do formal tests, such as the Shapiro-Wilk test, to detect normlity that may ruin the results of a two-sample t test.

If data have two independent samples from obviously non-normal populations, one should turn to nonparametric tests. We discuss two widely applicable ones: Mann-Whitney-Wilcoxon 2-sample rank-sum test, and a two-sample permutation test.

Note: As an illustration of the robustness of t tests, a one-sided Welch (separate-variances) t test of $H_0: \mu_ = \mu_2$ vs $H_a: \mu_1 < \mu_2$ in R gives P-value 0.0277. (Sample sizes are too small for the Shapiro-Wilk test to detect that the data are no normal, even though generated as gamma.)

Two-Sample Wilcoxon test. There is no assumption that the data are normal, and the test is intended to detect a 'shift' in the location of the population distribution between training and test data. Data are assumed continuous, so the (rounded) data analyzed should have few, if any ties. The null hypothesis we test is that there is no difference between population medians (no shift) $H_O:\eta_1 = \eta_2$ against the one-sided alternative is $H_a: \eta_1 < \eta_2.$

Here are fake samples x (say, patients were administered a placebo) and y (patients administered a drug) generated in R statistical software from non-normal populations, along with numerical and graphical descriptions, and a 2-sample, one-sided Wilcoxon test that find a significant difference in population medians at the 5% level but not at the 1% level.

set.seed(228)
x = rgamma(25, 5, .3);  y = rgamma(30, 5, .3)+5   # note different sample sizes
ummary(x);  summary(y)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
5.46   12.45   18.41   17.40   21.97   31.57
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
8.093  16.710  21.140  22.260  27.570  45.110
boxplot(x, y, col="skyblue2", pch=19)


wilcox.test(x, y, alte="less")

Wilcoxon rank sum test

data:  x and y
W = 262, p-value = 0.02839
alternative hypothesis: true location shift is less than 0


Permutation test. If there are many ties in the data or if the difference between the two groups may be more complicated than just a shift in location, it is sometimes better to do a permutation test.

As a simple example of a permutation test, suppose you had only four observations in each group, and that all of the drug observations exceed any of the placebo ones. Assuming independence of the groups and no difference between them, there would be only one chance in ${8 \choose 4} = 70$ of such an extreme outcome, thus the you would reject the null hypothesis that the two groups are from the same population with a significance level of $1/70.$

As sample sizes get larger, combinatorial analysis to find the null distribution and the P-value can become intractable. Suppose we want to know if there is a difference in population means. We begin by finding the observed difference $D = \bar X - \bar Y$ between the sample means.

Under the null hypothesis, it makes no difference whether an observation comes from the training or test group. We can get a good approximation of the permutation distribution by randomly switching group 1 and group 2 labels, and computing the difference $D^* = \bar X^* - \bar Y^*$ for each random re-assignment. We do this many times (using a different permutation of labels each time), then we find the P-value of the permutation test by seeing where the observed difference $D = \bar X - \bar Y$ falls in the approximated permutation distribution.

Here is how this test might look in R statistical software (where suffix prm replaces superscript $*$'s above). For the run shown the P-value is 0.016, again significant at the 5% level, but not at the 1% level. The graph below shows the simulated permutation distribution of $D^*.$ The P-value is the area of the histogram to the left of the vertical red bar at the observed value $D = \bar X - \bar Y = -4.853.$

Different runs (different seed or unspecified seed) would give slightly different P-values, but not enough different to change the decision to reject.

set.seed(228)  # same seed as above so same fake data
x = rgamma(25, 5, .3);  y = rgamma(30, 5, .3)+5
d.obs = mean(x) - mean(y)
dta = c(x, y); gp = c(rep(1, length(x)), rep(2, length(y)))
set.seed(218)  # change seed for different aprx null dist'n
m = 10^5;  d.prm = numeric(m)
for(i in 1:m) {
gp.prm = sample(gp)
d.prm[i] = mean(dta[gp.prm==1])-mean(dta[gp.prm==2]) }
mean(d.prm < d.obs)
## 0.01643   # simulated p-value of permutation test

hist(d.prm, prob=T, col="skyblue2", main="Permutation Dist'n of D")
abline(v=d.obs, lwd=2, col="red")
`

Reference: For more on 2-sample permutation tests see Sect. 4 od this afticle by L. Eudey et al. (2010).

For discrete distributions, this two-sample hypothesis testing question is very similar to the problem of closeness (or equivalence) testing considered in the property testing (distribution testing) community in theoretical computer science:

Given i.i.d. samples from two unknown distributions $p,q$ over a known discrete domain of size $n$, and a distance parameter $\varepsilon \in (0,1]$, distinguish with probability at least $9/10$ (or, more generally, $1-\delta$) between (i) $p=q$ and (ii) $d_{\rm TV}(p,q) > \varepsilon$

where $d_{\rm TV}$ is the total variation distance (or statistical distance) defined as $d_{\rm TV}(p,q)= \sup_{S} (p(S)-q(S)) = \frac{1}{2}\lVert p-q\rVert_1$. (Other distances, such as $\ell_2$ and Hellinger, have been considered.)

For this problem, tight (up to constant) bounds on the worst-case sample complexity have been obtained: $$n_1+n_2 = \Theta\!\left(\max\!\left(\frac{n^{2/3}}{\varepsilon^{4/3}}, \frac{\sqrt{n}}{\varepsilon^2}\right)\right)$$ are necessary and sufficient [1,2]. Further, in the unequal sample size case, a general trade-off between $n_1$ and $n_2$ is known as well [2,3].

At a high-level, these guarantees are obtained by a test which is essentially a suitable modification of a $\chi^2$-type test.

See e.g. [4] for a survey on distribution testing, and [5] for another focusing on one- and two-sample tests (written from a more traditional, Statistics point of view; see Section 4 (it mentions the continuous case as well)).

[1] Siu-on Chan, Ilias Diakonikolas, Paul Valiant, Gregory Valiant: Optimal Algorithms for Testing Closeness of Discrete Distributions. SODA 2014: 1193-1203

[2] Ilias Diakonikolas, Daniel M. Kane: A New Approach for Testing Properties of Discrete Distributions. FOCS 2016: 685-694

[3] Bhaswar B. Bhattacharya, Gregory Valiant: Testing Closeness With Unequal Sized Samples. NIPS 2015: 2611-2619

[4] Clément L. Canonne: A Survey on Distribution Testing: Your Data is Big. But is it Blue? Electronic Colloquium on Computational Complexity (ECCC) 22: 63 (2015)

[5] Sivaraman Balakrishnan and Larry Wasserman: Hypothesis Testing for High-Dimensional Multinomials: A Selective Review. arXiv:1712.06120 (2017)

• Thanks for posting this. This method is new to me. – BruceET Mar 2 '18 at 20:07
• Also relevant is this very recent survey by Balakrishnan and Wasserman (2017): Section 4 tackles two-sample testing. – Clement C. Mar 2 '18 at 20:14