# reason of the definition of the covariance

The covariance of two random variables $X$ and $Y$ is defined to be $${\rm Cov}(X,Y) = E[(X-E[X])(Y-E[Y])].$$ I don't understand it, if someone could explain me this please. Why does this value tell us of some relation about $X$ and $Y$?

-
Simple correlation of functions $f$ and $g$ is $\int_X fg \, \mathrm{d}\mathbb{P}.$ Here we just subtract their mean values before integrating to obtain desirable properties, that is, so that adding constants wouldn't change covariance. –  Alexei Averchenko Jul 4 '11 at 1:26
@Alexei: Simple or not, the correlation of $f$ and $g$ is not the integral of $fg$. –  Did Jul 4 '11 at 7:59
@Didier: I didn't mean correlation coefficient. –  Alexei Averchenko Jul 4 '11 at 9:51
@Alexei: Sorry, I cannot read your mind but only what you write. –  Did Jul 4 '11 at 9:58

Consider the special cases where $X=Y$, and where $X=-Y$. In the first case, $X-E[X]$ and $Y-E[Y]$ will always have the same sign, so the product will always be positive, as will its expectation; in the other case, the one will be positive when the other is negative, and vice versa, so the product and the expectation will be negative. So this makes covariance positive if the things vary together, and negative if they vary oppositely.

-

What might interest you is the correlation coefficient $$\rho_{XY} = {{\rm Cov}(X,Y)\over \sigma_X \sigma_Y.}$$ If $\rho_{XY} = 1$, $X$ and $Y$ are linearly related via a linear relation with a positive slope. If $\rho_{XY} = -1$, $X$ and $Y$ are linearly related via a negative slope. If $X$ and $Y$ are independent $\rho_{XY} = 0$ (beware: converse is NOT true).

-

Assume the X and Y - data in an array of 2 rows and n columns.

Now assume, that each column defines a new dimension in the n -dimensional euclidean space. For instance, if n is 2, then X as well as Y has two coordinates and you can draw vectors in the plane from the origin to the endpoints X and Y. You can see the angle between the vectors physically if you draw this on your paper. If n=3 , you have a threedimensional space and the two vectors are located in the space, but still you can imagine the vectors and their angle.

Generalize this to n-dimensions. The relevant relation for us is the angle between the vectors. But this angle can be expressed by the cosine and the formula of the cosine is just that of the correlation (rescaled covariance) and after the mean was subtracted.

There are two more aspects relevant:

a) The removal of the mean is just convention: with the idea of correlation/covariance we want the mass of simultane deviation of the mean/expected value - so the origin of the vectors is translated to the overall mean

b) if the observations/measures in X resp in Y are statistically "not independently" sampled, then we can assume the euclidean space as non-orthogonal, meaning with some way of skew/oblique axes. This indicates also, how such data could then be "orthogonalized" to remove that dependency (at least in principle)

-