# Covariance of random numbers not near zero

I'm learning about making an efficient bounding volume for a point cloud and this touches on statistics: an area of mathematics I'm very unfamiliar with.

The book talks about how to compute covariance matrices. It explains that the off-diagonal entries represent the correlation between each pair of x and y (and z if 3d) coordinates with an entry of zero indicating no correlation between the coordinates used to compute the entry:

I also found a video on YouTube about covariance which I liked and around 14:20 into the video the claim is made that the covariance will be near or equal to zero.

I decided to write some code to test this near zero covariance claim for uncorrelated variables. My code generates 5000 2D points with each $x$ and $y$ entry randomly a member of $[0,500]$. The mean point is near $(250,250)$ as expected. $\text{cov}(x,y)$ however is not typically close to zero. Examples of covariances for $x$, $y$ I'm computing are: $505.08$, $316.05$, $191.52$, $-148.70$. These would be the off-diagonal entries in the covariance matrix. How come the covariance isn't near zero?

I've triple-checked the code, but haven't completely ruled out the possibility of coding error. The C++ code can be found here.

• As a quick check: What covariance matrix do you get for the set of points $\{(0,0,0),(0,1,0),(1,0,0),(1,1,0)\}$? (or the same thing without the third coordinates...) Commented Jul 3, 2014 at 15:06
• @EricTowers I get $\begin{bmatrix} \frac{1}{4} & 0 & 0\\ 0 & \frac{1}{4} & 0\\ 0 & 0 & 0 \end{bmatrix}$ implying a covariance of zero. Commented Jul 4, 2014 at 13:48
• @PeteUK sample covariances need not be almost zero if the diagonal elements are large. See my post...I think you'll see that your results are correct, but you need to use the correlation coefficient to interpret what "small" means.
– user76844
Commented Jul 6, 2014 at 5:31
• @Eupraxis1981 Thanks for the comment. I haven't gone through your answer yet but what you just said helps a lot. I'll look at your answer tonight or tomorrow and let you know if I get it! Commented Jul 6, 2014 at 9:24

"Near zero" is a very vague term and needs some context. Near compared to what? This is the problem with the covariance: $Cov(X,Y) = 1$ could be large or small, depending on the variance of X and Y. This is why we have correlation: it is a normalized covariance:

$Corr(X,Y)=\frac{Cov(X,Y)}{\sqrt{Var(X)Var(Y)}}$, which corrects for the overall variability of X and Y.

Since you are modelling your x,y points as uniform random variables between 0 and 500, which will each have a variance of $\sigma^2_{X\; \text{or}\;Y}=\frac{500^2}{12} \approx 20,833$ and standard deviation $\sigma_{X\; \text{or}\;Y}=\sqrt{\sigma^2_{X\; \text{or}\;Y}} = \sqrt{20,833}\approx144$, your observed covariances of 505.08, 316.05, 191.52, −148.70 need to be divided by $\sqrt{Var(X)Var(Y)} = \sqrt{20,833^2} = 20,833!$ Doing this you get the following sample correlations:

$\{0.024,0.015,0.009,-0.007\}$ -- small correlations indeed! So, I think a better thing to say is that uncorrelated variables will have correlations near 0, not their covariances.

• Can you explain in more detail $\frac{500^2}{12}$, especially where the 12 in the denominator comes from? Commented Jul 7, 2014 at 14:32
• You were simulating random variables that are uniformly distributed. A uniform random variable on $[A,B]$ has variance$\frac{(B-A)^2}{12}$. This is a formula that you can look up (or derive from $\frac{1}{B-A}\int\limits_A^B (x-\frac{A+B}{2})^2 dx$
– user76844
Commented Jul 7, 2014 at 14:36
• Thanks for your answer. When time permits I'll learn more thoroughly about correlations and (co)variances and then your answer will make sense "from the ground up". The main thing I've learnt is that the covariance is difficult to interpret by itself, but the correlation (normalised covariance) is far easier to read (0 = no correlation, 1 = perfectly correlated). Commented Jul 8, 2014 at 12:10
• @PeteUK That's exactly right (don't forget that both Cov. and Corr. can be negative too)! We have the correlation precisely because covariance is not a measure of the strength of an association. Also, in your readings, keep in mind that covariance and correlations are linear measures of association, so seek counterexamples where you have a strong relationship between two variables yet the covariance or correlation is zero (think about how you could transform such cases so correlation will work). Also look at non-parametric measures of association. Good luck with your studies!
– user76844
Commented Jul 8, 2014 at 12:18