# Statistics and confidence - intervals

An account on server A is more expensive than an account on server B. However, server A is faster. To see whether it's optimal to go with the faster but more expensive server, a manager needs to know how much faster it is . A certain computer algorithm is executed 20 times on server A and 30 times on server B with the following results,

                  Server A  Server B
Sample means      6.7 min   7.5 min
Sample std. dev.  0.6 min   1.2 min


A 95% confidence interval for the difference $\mu_{1} - \mu_{2}$ between the mean execution times on server A and server B is [-1.4,-0.2] . Is there a significant difference between the two servers?

(a) Use the confidence interval above to conduct a two-sided test at the 1% level of significance.

(b) Compute a p-value of the two-sided test in (a).

(c) Is server A really faster? How strong is the evidence? Formulate the suitable hypothesis and alternative and compute the corresponding p-value.

-
What are your thoughts on the exercise? – Stefan Hansen May 27 '13 at 7:44
Well I guess I should find $H_0$ with $\mu_{0}$ respectively $H_A$ with $\mu$ and then use the formula $T= \overline{x} - 2 / \sigma / \sqrt{4}$ I suppose. Then check if the value from T is in the acceptance range or not, acceptance being $-t_{\alpha / 2}$ and $t_{\alpha / 2}$ but that's like all I've got. – Contourette. May 27 '13 at 12:24
Well, before anyone can say anything, you need to make assumptions. Can these be considered as two independent normal distributed samples? Do the two samples have the same standard deviation? – Stefan Hansen May 27 '13 at 15:51
I'm not following you.. and no they don't have the same standard dev – Contourette. May 27 '13 at 16:43
The fact that you're talking about $t_{\alpha/2}$ from a $t$-distributions means that there are some kind of normal distribution assumption. – Stefan Hansen May 27 '13 at 17:28

Setup: Assume that both samples $$\text{Server A:}\quad x_{11},\ldots,x_{1n_1},\\ \text{Server B:}\quad x_{21},\ldots,x_{2n_2}.$$ are i.i.d. observations coming from a normal distribution with unknown mean and variance. Here $n_1=20$ and $n_2=30$. This can in short be formulated as $x_{ij}$ are realizations from independent random variables $X_{ij}$ and $$X_{ij}\sim \mathcal{N}(\mu_i,\sigma_i^2),\quad j=1,\ldots,n_i,\;\;i=1,2.$$

Let $\bar{X}_1$ and $\bar{X}_2$ denote the sample means, i.e. $$\bar{X}_i=\frac1{n_i}\sum_{j=1}^{n_i}X_{ij}\sim \mathcal{N}\left(\mu_i,\frac{\sigma_i^2}{n_i}\right),\quad i=1,2.$$ Then $\bar{X}_i$ is estimating the true mean $\mu_i$, and the difference of means $\mu_1-\mu_2$ is of course estimated by $\bar{X}_1-\bar{X}_2$.

Assumption of Equal Variances: We furthermore assume that the variance in the two groups are equal, i.e. $\sigma_1^2=\sigma_2^2$ and we denote the common variance by just $\sigma^2$. If $S_i^2$ denotes the sample variance of the $i$th sample, i.e. $$S_i^2=\frac{1}{n_i-1}\sum_{j=1}^{n_i}\left(X_{ij}-\bar{X}_i\right)^2,\quad i=1,2,$$ then $$S^2=\frac{(n_1-1)S_1^2+(n_2-1)S_2^2}{n_1+n_2-2}$$ is the estimator of the common variance $\sigma^2$. Now one can show that $$T=\frac{(\bar{X}_1-\bar{X}_2)-(\mu_1-\mu_2)}{\sqrt{S^2\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}\sim t(f)\tag{1}$$ where $f=n_1+n_2-2$. Knowing that $T$ follows a $t$-distribution with $f$ degrees of freedom is exactly what we need in order to find confidence intervals and perform tests.

$99\%$ confidence interval: A $99\%$ confidence interval is obtained by manipulating $(1)$ and we obtain that $$99\%=P\left(\underbrace{\bar{X}_1-\bar{X}_2-C\cdot t_{0.995}(f)}_{\text{lower bound}}\leq\mu_1-\mu_2\leq \underbrace{\bar{X}_1-\bar{X}_2+C\cdot t_{0.995}(f)}_{\text{upper bound}}\right)$$ where $t_{0.995}(f)$ is the $99.5\%$ fractile of a $t$-distribution with $f$ degrees of freedom and $$C=\sqrt{S^2\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}.$$

Hypothesis testing: We wish to test the statistical hypothesis that the two means are equal meaning that Server A and Server B are equally fast. The hypothesis is written as $$H\!:\mu_1=\mu_2$$ and is of course equivalent to testing that $\mu_1-\mu_2=0$. Under the hypothesis, $(1)$ becomes $$T=\frac{\bar{X}_1-\bar{X}_2}{\sqrt{S^2\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}\sim t(f)\tag{2}$$ and as numerically large values are critical for the hypothesis we obtain the $p$-value $$p=2\left(1-F_{t(f)}(|T|)\right),$$ where $F_{t(f)}(\cdot)$ denotes the CDF of a $t$-distribution with $f$ degress of freedom. We accept $H$ if $p>0.01$ and reject it if $p<0.01$. Checking whether $p>0.01$ actually corresponds to checking whether $\bar{X}_1-\bar{X}_2$ is in the $99\%$ confidence interval or not.

Model Checking: This answer is based on some crucial assumption which, of course, have to be checked. That this is indeed two i.i.d. samples from normal distributions should of course be checked. Futhermore, the assumption of equal variances should be checked. This can be checked by performing an $F$-test.

-

a) $\overline{X_1}=6.7, s_1=0.6 , n_1=20$ $\overline{X_2}=7.5 , s_2=1.2, n_2=30$

$H_0$: $\mu_1=\mu_2 , \mu_1-\mu_2=0$ $H_A$: $\mu_1 \neq \mu_2$

df (degrees of freedom)= 45

t(critical values)=$\pm 2.686$

T(test statistic)=$\frac{\overline{X_1}-\overline{X_2}}{std(\overline{x_1}-\overline{X_2})}$

std=$\sqrt{V(\overline{X_1}-\overline{X_2})}=\sqrt{V(\overline{X_1})-V(\overline{X_2})}=\sqrt{\frac{0.6^2}{20}+\frac{1.2^2}{30}}=\sqrt{0.018+0.048}=\sqrt{0.66}=0.256$ $\overline{X_1}-\overline{X_2}=-0.8$

$T=\frac{-0.8}{0.256}=-3.112$

$T=-3.112$ is not in the acceptance range of -2.686 and 2.686 so we reject the null hypothesis

b) $p-value= P(T>3.112)+P(T<-3.112)= 2*0.001=0.002$

$0.002<0.01$ therefore the results show that there is a highly significant difference between the two servers.

c)$H_0 : \mu_1=\mu_2$

$H_A:\mu_1<\mu_2 , \mu_1-\mu_2<0$

$T=-3.112$ , $\alpha=0.01$ , $t_\alpha=2.4$ $p-value=P(T< -3.112)= 1-(1-valueOf(t_\alpha))=1-(1-0.001)=1-0.999=0.001$ $0.001<0.01$ therefore there is highly significant evidence and we reject $H_0$

Conclusions: From a), b) and c) we can say that there is highly significant evidence that there is a difference between the two servers and that server A is faster than server B.

-
How do you get $\text{df}=45$? Furthermore, unless you assume that the variances in the two groups are equal, your test statistic does not follow a $t$-distribution. See e.g. this. – Stefan Hansen Jun 7 '13 at 10:45