http://en.wikipedia.org/wiki/Covariance_matrix

Cov(Xi,Xj) = E((Xi-Mi)(Xj-Mj))


Is the above equivalent to:

(Xi-Mi)(Xj-Mj)


I don't understand why the expectancy of (Xi-Mi)(Xj-Mj) would be different than just (Xi-Mi)(Xj-Mj)

Let's say I have two sets of data:

Set 1: 1,2,3 avg: 2

Set 2: 4,5 avg: 4.5

Is the following a covariance matrix?

(1-2)*(4-4.5) , (2-2)*(4-4.5) , (3-2)*(4-4.5)
(1-2)*(5-4.5) , (2-2)*(5-4.5) , (3-2)*(5-5.5)


I'm reading online and it seems like a covariance matrix composed of two data sets should be a 2x2 matrix, but in this case, I have a 2x3 matrix. How do I go from what I have to the correct 2x2 matrix?

-
The $X_i$ are random vairables, so $(X_i-M_i)(X_j-M_j)$ is a random variable, and $Cov(X_i,X_j)$ is its expectation value. What do you not understand? – Rasmus Feb 1 '11 at 23:42
Your example "sets of data" does not make sense. Each sample-observation should include a fixed number of features, i.e., each must be of the same size. See the example of the 5 students with 3 courses here :stattrek.com/matrix-algebra/variance.aspx There, you have a X of dimension 3 (tests results in the three courses, for a given student), and you have 5 samples. The covariance matrix would be 3X3 – leonbloy Feb 2 '11 at 3:12

It's like asking if $X$ is equivalent to $E(X)$.

$X$ is (asssumed to be) a random variable, it can take several values according to some probability law. In contrast $E(X)$ is the expected value of $X$ - so it's not a random variable which takes different values in different tries, it's a constant number. So they are conceptually different entities (they coincide only in the degenerate-trivial case in which the random variable only takes one value).

The same applies to the covariance matrix. In your formula $X_i$ is a random variable, so the product is also a random var (because it's a function of random variables!). The expectation operator "takes out" the randomness, and so the covariance matrix is a constant matrix.

-
Can you comment on my addendum in the OP? Thanks :) – Soo Feb 1 '11 at 23:54
You are confusing the dimensionality of the data with the size of your sample. – leonbloy Feb 2 '11 at 3:05

There is a difference between a random variable and its expectation. Take a standard fair six sided die. It can take any of the values {1,2,3,4,5,6}, but its expectation is 3.5.

Similarly $(X_i - \mu_i)(X_j - \mu_j)$ is a random variable with several possible values while $\mathrm{E}[(X_i - \mu_i)(X_j - \mu_j)]$ is a number depending on the joint distribution of $X_i$ and $X_j$.

-

The covariance matrix is $Cov(X_i,X_j)=E[(X_i-\mu_i)(X_j-\mu_j)]$ Each $\mu_i=E[X_i]$. the expected value of variable $i$. If your variables are independent, the off-axis covariance terms are zero because $E[(X_i-\mu_i)(X_j-\mu_j)]=E[(X_i-\mu_i)]E[(X_j-\mu_j)]$ and both terms on the right are zero. The diagonal terms are just the variances of each variable, as you should be able to see from the definition. You are right that the covariance matrix for your data should be 2x2, as the dimension comes from the number of variables, not from the number of data points.

-