3
$\begingroup$

I need to test random numbers generators in 1, 2 and 3 dimensions. Criteria of test is that generated numbers are from uniform distribution. I am doing this using Matlab. I'm told to use Chi-2 test statistic $$\chi^2=\sum^B_{i=1}\frac{(expected(i)-observed(i))^2}{expected(i)}$$ where $B$ is number of bins, $expected(i)=\frac{N}{B}$ for all $i$ is expected number of elements in $i_{th}$ bin and $N$ is number of generated numbers.

What I have done is that I generated

  • $N\times 1$ matrix of random numbers for $1D$
  • $N\times 2$ matrix of random numbers for $2D$
  • $N\times 3$ matrix of random numbers for $3D$

Then, I set a number of bins (B) for each component of matrix elements and I created zero matrices of sizes

  • $1\times B$ for $1D$
  • $B\times B$ for $2D$
  • $B\times B\times B$ for $3D$

and, using loops, I counted how many generated numbers are in which bin and those were, actually, values $observed(i)$. At this point I am able to calculate $\chi^2$ test statistic for all 3 dimensions.

I know how to read $\chi^2$ table but I don't know how to check significance using Matlab. I'm also not sure about degrees of freedom, but I think it's $B-1$ for $1D$, $BB-1$ for $2D$ and $BBB-1$ for $3D$ since we don't have to estimate any parameter for uniform distribution. I would appreciate some help. Thanks!

$\endgroup$
2
$\begingroup$

That all sounds right; looks like you're almost done. All you need to do now is call chi2cdf(x,v,'upper') where $x$ is your $\chi^2$ test statistic and $v$ is your degrees of freedom; this will output the P-value for the test.

$\endgroup$
  • $\begingroup$ Thank you! While I was waiting for some help here, I found that p-value is 1-chi2cdf(x,v). Now I am a bit confused. Can you tell me when we use chi2gof? $\endgroup$ – user23709 Mar 14 '15 at 16:31
  • $\begingroup$ Yes, 1-chi2cdf(x,v) is the same as chi2cdf(x,v,'upper'), although the former is more prone to floating-point roundoff error and hence may give a less accurate answer if the P-value is extremely small. The way you have solved this problem, you have calculated the $\chi^2$ test statistic manually. chi2gof is a function which can do this for you; if you pass it the appropriate options, it should give you the same answer as before. $\endgroup$ – Brent Kerby Mar 14 '15 at 16:41
  • $\begingroup$ I wrote a code but I think it doesn't work correctly. When I generate numbers from uniform distribution, it says that distribution of uniformly distributed numbers is consistent with uniform, which is expected. But it also happens when I generate numbers from normal distribution. Here is the code on google drive (docs.google.com/document/d/…). I would appreciate if you would check it. You are also allowed to comment it. $\endgroup$ – user23709 Mar 17 '15 at 10:42
  • $\begingroup$ All right, I've spotted a bug and commented on it. $\endgroup$ – Brent Kerby Mar 17 '15 at 14:04
  • $\begingroup$ Thanks. I've changed it. Maybe it's not still correct. I put a code for numbers generated from N(0,1) distribution and I tested it for uniform distribution. For 1D I got that it's not consistent with uniform distribution, but for 2D and 3D I got that it is. $\endgroup$ – user23709 Mar 17 '15 at 15:16
1
$\begingroup$

Just to expend on it and give the all the code.

Setting parameters

N=100; % sample size
a=0; % lower boundary
b=1; % higher boundary

Sample N uniformly distributed values between a and b. And in the second line add some bais to make it not uniform if you want to test the code.

x=unifrnd(a,b,N,1);
%x(x<.9) = rand(sum(x<.9),1);

Using chi2gof

As described here, with chi2gof, you can't use the 'cdf of the hypothesized distribution' and need to specified the bins, the edges and the expected values.

nbins = 10; % number of bin
edges = linspace(a,b,nbins+1); % edges of the bins
E = N/nbins*ones(nbins,1); % expected value (equal for uniform dist)

[h,p,stats] = chi2gof(x,'Expected',E,'Edges',edges)

Using chi2cdf

With this function you need to supply the chi-squared test statistic, $\displaystyle \chi ^{2}$ which can be computed with the function histogramm:

h = histogram(x,edges);
chi = sum((h.Values - N/nbins).^2 / (N/nbins));
k = nbins-1; % degree of freedom
chi2cdf(chi, k)

Note, that if you don't use the edges to compute the number of value per bins, histogramm will choose them from the lower value to the highest and therefore the final score will be different than with chi2gof

Quick Theory recall

Just to recall how to interpret the final value, few definition:

  • The null hypothesis ($H_0$) is that the data x are coming from a uniform distribution.
  • Pearson's chi-squared test is testing if you can safely reject the null hypothesis, i.e. "Can I say that x is not a coming from a uniform distribution ? "
  • Pearson's cumulative test statistic $\displaystyle \chi ^{2}$ is a measure of the error between observations and expected value $$\displaystyle \chi ^{2} = \sum_{i=1}^N \frac{(O_i-E_i)^2}{E_i}$$
  • Chi-squared distribution is the distribution that the Pearson's cumulative test statistic $\displaystyle \chi ^{2}$ would follow according $H_0$ (i.e. if the observation are coming from a uniform distribution)
  • The p-value is the probability of obtaining a worst result than what was observation, when the null hypothesis is true. That is, the probability that, if we randomly draw a dataset y in a uniform distribution, the error (or Pearson's cumulative test) will be equal of higher than the actual observation x (=worst case).
  • So, we can reject $H_0$ if p is lower than a significant level $\alpha$. That is, for small value of p, we can safely say that x is not coming from a uniform distribution.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.