What statistical test should be used to evaluate the efficiency of some treatment?

Question

We have two group of people (one of $n$ people, the other of $m$ people) that go through the same entertainment experience (think about a day at the amusement park or something like that).

One group gets a special treatment over the other group at some moment during the day (a free drink for example). At the end of the day, each participant gives a satisfaction grade (an integer) $X_1, \cdots, X_n, Y_1, \cdots, Y_m$ that ranges from $0$ to $5$.

We would like to know if the special treatment affects the overall satisfaction of the experience.

I guess that we have to test if the means of the satisfaction of the two groups are equal ? What test should be used ?

Answer 1

Fake data for illustration. Suppose there are $n = 40$ subjects in the X-group and $m = 60$ in the Y-group, with tallied satisfaction scores as follows:

table(x)
x
 0  1  2  3  4  5 
 3  5 16  9  6  1 

table(y)
y
 0  1  2  3  4  5 
 4  5 12  3 20 16 

Nonparametric rank-based Wilcoxon test. Then a 2-sample Wilcoxon rank sum test of $H_o: \eta_1 = \eta_2$ against $H_a: \eta_1 \ne \eta_2.$ (from R statistical software) gives results as follows:

wilcox.test(x, y)

        Wilcoxon rank sum test with continuity correction

data:  x and y 
W = 737, p-value = 0.0008556
alternative hypothesis: true location shift is not equal to 0 

So for my fake data there is a strongly significant difference in the two population means.

Boxplots with notches. Because boxplots are fundamentally based on order statistics (valid for ordinal data), I have made side-by-side boxplots of the data. The style shown here has 'notches' in the sides of the boxes, which show nonparametric confidence intervals (CIs) for the two group population medians. Roughly speaking these CIs are calibrated so that non-overlapping CIs indicate a significant difference in medians. (This is not as powerful a test as the Wilcoxon test, but the effect for the current dataset is strong enough to show significance.)

If this is for a report or for publication, it might be worthwhile to show boxplots--mentioning sample sizes, but possibly without the CIs (which might be a bit technical for a non-statistical audience). There are few effective graphical displays for categorical data, so it is worthwhile noting possibilities.

Note: Another possible test would be to make a $2 \times 6$ matrix of counts, with rows for the X and Y groups and columns for the opinion scores. Then do a chi-squared test of independence. I would illustrate this, but I see that @Remy has concurrently posted an Answer (+1) along those lines. A finding of association rather than independence might be all you need, but the Wilcoxon test inherently suggests the 'direction' of the effect.

Addendum (per request in Comment). Here is R code I used to make the fake data. I did not set a seed, so each run of the code gives a different simulated dataset. [Notice that elements of the prob argument need not sum to unity; before use, R normalizes the vector of proportions to get probabilities summing to $1.]$

x = sample(0:5, 40, rep=T, prob=c(1,2,3,3,2,1))
y = sample(0:5, 60, rep=T, prob=c(1,1,2,2,3,3))

Answer 2

Since the options are integers, this is a likert scale so the data is ordinal. You can use a Chi-Square Test for Independence to test the hypotheses

$$H_0: p_{i0}= p_{0j} \text{ for all cells } (i,j)$$

$$H_a:\exists(i,j) \text{ such that } p_{i0} \neq p_{0j}$$

or more simply

$$H_0: \text{group and satisfaction level are independent}$$

$$H_a: \text{group and satisfaction level are associated}$$

We have

$$p_{i0}=\frac{n_{i0}}{n}, p_{0j}=\frac{n_{0j}}{n}$$

The assumption is that all of the expected cell counts $\geq 5$.

We have

$$X^2=∑_{all cells}\frac{(n_{ij}-E_{ij})^2}{E_{ij}}$$

where

$$E_{ij}=\frac{(n_{i0} n_{0j})}{n}$$

and

$$X^2\sim\chi_{(r-1)(c-1)}^2$$

Using Bruce's fake dataset, the test can be ran in R using:

library(reshape)

df <- data.frame(Rating = c("0","1","2","3","4","5","0","1","2","3","4","5"),
                 Group= c("x","x","x","x","x","x","y","y","y","y","y","y"),
                 INTERACTIONS = c(3,5,16,9,6,1,4,5,12,3,20,16),
                 stringsAsFactors=FALSE)

df <- melt(df,id.vars=c("Rating","Group"))    
df <- cast(df,formula=Rating~Group)
df <- replace(df,is.na(df),0)

chisq.test(df)

which returns

Pearson's Chi-squared test

data:  df
X-squared = 21.342, df = 5, p-value = 0.0006981

so we have very strong evidence that the two groups having differing satisfaction levels.

Answer 3

Comment. A closer look at the chi-squared test:

Here is a somewhat simplified way to do the chi-squared test for independence.

MAT = matrix(c(3,5,16,9,6,1,  4,5,12,3,20,16), byrow=T, nrow=2)
tst.inf = chisq.test(MAT);  tst.inf

        Pearson's Chi-squared test

data:  MAT 
X-squared = 21.3417, df = 5, p-value = 0.0006981

Warning message:
In chisq.test(MAT) : Chi-squared approximation may be incorrect

The warning message may be on account of one or more expected counts that are 'too small'. The object tst.inf contains more information than is routinely shown. In particular we can look at the expected counts:

tst.inf$exp
     [,1] [,2] [,3] [,4] [,5] [,6]
[1,]  2.8    4 11.2  4.8 10.4  6.8
[2,]  4.2    6 16.8  7.2 15.6 10.2

We see that the first two columns contain undesirably small expected counts. In view of the very small P-value this is not likely a serious problem.

If one thinks it is a problem, there are two 'cures': (1) Combine the first two response levels to get a new matrix of observed counts, and run the test again. (2) Let R simulate the correct P-value for the original observed values, rather than using the chi-squared approximation; in effect, this is a structured permutation test based on matrices with the correct marginals. (1) Is routine; I will show (2):

chisq.test(MAT, sim=T)

    Pearson's Chi-squared test with simulated p-value (based
    on 2000 replicates)

data:  MAT 
X-squared = 21.3417, df = NA, p-value = 0.001499

The simulated P-value is larger than the questionable one above, but it is still well below 0.05, so rejection at the 5% (or even the 1% level) is warranted.