5
$\begingroup$

I'm trying to understand a geometric interpretation of covariance as explained in this lecture pdf, which states:

If X, Y are two random variables of zero mean, then the covariance Cov[XY ] = E[X · Y ] is the dot product of X and Y. The standard deviation of X is the length of X. The correlation is the cosine of the angle between the two vectors.

I can understand how, given a zero mean, the standard deviation of X is the length of X. What I don't understand is how the the covariance of X and Y is equal to it's dot product.

Specifically, the covariance is defined as:

$Cov(X,Y) = E[(X - \mu_X) (Y - \mu_Y)]$

Which with we can write as:

$Cov(X,Y) = \sum\limits_{i}\sum\limits_{j} p_{ij} (x- \mu_X) (y-\mu_Y)$

And since for two centered random variables the means are zero, and the probability is uniform, we can rewrite the formula as:

$Cov(X,Y) = \dfrac{1}{N} \sum\limits_{i}\sum\limits_{j} x y$ (Eqn 1)

The dot product, in contrast consists of a single summation:

$dot(X,Y) = \sum\limits_{i} x_i y_i$ (Eqn 2)

Note that the $\frac{1}{N}$ factors out of Eqn 1 when I divide by the standard deviations of X and Y. So I almost can derive the dot product from the covariance, but the covariance still contains a double summation and the dot product has a single element-wise summation. What am I missing?

UPDATE

I don't have the answer, but I realize that this formula:

$Cov(X,Y) = \sum\limits_{i}\sum\limits_{j} p_{ij} (x- \mu_X) (y-\mu_Y)$

... where N is the dimension of the vectors, is not equal to Eqn 1:

$Cov(X,Y) = \dfrac{1}{N} \sum\limits_{i}\sum\limits_{j} x y$ (Eqn 1)

Since if $p_{ij}$ equals $\dfrac{1}{N}$, it would sum to be greater then one in the double summation. $p_{ij}$ only equals $\dfrac{1}{N}$ when it's a single summation, which maps easily to the dot product. I've posted a new question here to try and address this confusion in isolation, which I hope in turn will clarify this derivation problem.

$\endgroup$
3
  • $\begingroup$ I may be wrong, but I think the author is stating the covariance to be the dot product, in the sense that it is akin to the dot product. If you take this view, then note that $\cos \theta = \frac{u \cdot v}{|u||v|}$ parallels $r = \frac{Cov(X,Y)}{\sigma_X \sigma_Y}$. This is a commonly known analogy or connection. See johndcook.com/blog/2010/06/17/covariance-and-law-of-cosines $\endgroup$ Commented Jan 14, 2021 at 3:29
  • $\begingroup$ @OsamaGhani, I believe that it's not just an analogy, I'm able to derive XY/|X||Y| = Cov(X,Y)/(sdX, sdY) if we assume the data in X and Y are mean centered vectors. Then the standard deviation is equal to the magnitude of the vectors, and Cov(X, Y) is equal to the dot product if we assume Cov(X,Y)=∑ipi(xi−μX)(yi−μY). I've updated my question to show that where this derivation is causing confusion is that sometimes Cov(X,Y)=∑i∑jpij(xi−μX)(yj−μY), which doesn't map to the dot product. Any idea what I'm getting wrong with these two, seemingly conflicting, definitions of the covariance? $\endgroup$
    – saeranv
    Commented Jan 14, 2021 at 4:07
  • $\begingroup$ if X and Y have zero mean, then the trace of their covariance is their inner product. $\endgroup$
    – Car Loz
    Commented Nov 21, 2022 at 16:17

1 Answer 1

1
$\begingroup$

The notes seem to be giving a result before stating the interpretation on which the result is based. If you only read the first page of the notes, the statements about vectors don't make sense.

On the second page we start to get an explanation of what is really meant by the use of vector notation on the first page. We are supposed to consider a probability space $\Omega$ with $n$ elements ($n$ a positive integer), where each element has the same probability measure as any other. The elements are listed in sequential order from the $1$st element to the $n$th.

We are then supposed to consider a vector in $\mathbb R^n$ as a function that takes the sequential number of an element of the probability space as input, and gives a real number as output. This matches the definition of a real random variable over that probability space, which is a function that maps each element of the probability space to a real number.

Now let's consider something like the expected value of the square of a random variable $X$, that is, $E[X^2]$. The formula $E[X^2] = \sum_i x_i^2 p(x_i)$, where $p$ is the pmf of the random variable $X$ and the values of $X$ are represented by $x_i$, is not the only way to compute this expected value. For a finite probability space we can also compute the expected value by taking the sum of a different sequence of products, $$ E[X^2] = \sum_{\omega \in \Omega} \mu(\omega) (X(\omega))^2, \tag1 $$ where $\omega$ ranges over the elements of the probability space $\Omega$, $\mu(\omega)$ is the probability measure of the element $\omega,$ and $X(\omega)$ is the value of the random variable $X$ for input $\omega$.

The equation $$ E[X^2] = \frac1n \sum_{i=1}^n X_i^2 \tag2 $$ is just a rewriting of Equation $(1)$ for a probability space with $n$ elements of equal probability $\frac1n$, where $X_1,\ldots, X_n$ are the values of the random variable $X$ for the $1$st through $n$th elements of the probability space. The right-hand side of $(2)$ is also $$ \frac1n (X \cdot X) = \frac1n \lVert X\rVert^2 $$ if $X$ is represented by a vector as suggested on the second page of the notes.

Similarly, for $E[XY],$ the expected value of the product of random variables $X$ and $Y$, we can write $$ E[XY] = \sum_{\omega \in \Omega} \mu(\omega) X(\omega) Y(\omega). $$

For a probability space of $n$ elements of probability $\frac1n$ each, using vectors to represent $X$ and $Y,$ this translates to $$ E[XY] = \frac1n \sum_{i=1}^n X_i Y_i. \tag3$$

Note: this is not a double summation over $i$ and $j$ of terms in $X_i Y_j$, and there's a good reason for that. The summation in this formula was really over the elements of the probability space. Each index value $i$ picks out one element. Once you have picked an element of the probability space, each random variable's value is determined. So when we pick the $i$th element, the value of $X$ is $X_i$, the value of $Y$ is $Y_i$, and the value of $XY$ is $X_i Y_i$.

Now note that if we use vectors for $X$ and $Y$, the right-hand side of $(3)$ can also be written $$ \frac1n (X \cdot Y). $$

So the notes are literally correct when talking about a finite probability space with a uniform probability measure, with vectors in $\mathbb R^n$ interpreted as random variables over that probability space, except for one detail: they seem to have omitted the factor $\frac1n$.


A further note to explain why we sometimes use a single summation for covariance and sometimes a double summation. Suppose we have a probability space $\Omega = \{1,2,3,4,5\}$ with uniform probability measure (each element has probability measure $\frac15$). Suppose $X$ and $Y$ are random variables and that \begin{align} X(1) &= 11, & Y(1) &= 21, \\ X(2) &= 11, & Y(2) &= 22, \\ X(3) &= 12, & Y(3) &= 21, \\ X(4) &= 12, & Y(4) &= 23, \\ X(5) &= 13, & Y(5) &= 21. \\ \end{align} So we see that the possible values of $X$ are $x_1 = 11,$ $x_2 = 12,$ and $x_3 = 13,$ whereas the possible values of $Y$ are $y_1 = 21,$ $y_2 = 22,$ and $y_3 = 23.$ When we take pairs $(x_i,y_j)$ there are therefore $3\times 3 = 9$ possible pairs, and when we write $$ \sum_{i=1}^3 \sum_{j=1}^3 x_i y_j P(X=x_i, Y=y_j) $$ we have to add up $9$ terms in total from the double summation.

But actually, according to the definitions of $X$ and $Y$ above, there are only $5$ possible values of $(x_i,y_j)$:

$$ (11,21), (11,22), (12,21), (12,23), (13,21). $$

So the double summation is "wasteful" in that it counts several outcomes that have zero probability. The single summation $$ \frac15 \sum_{i=1}^5 X(i) Y(i), $$ on the other hand, counts each of the $5$ pairs of values from $X$ and $Y$ that can occur and does not bother to count any of the zero-probability pairs. So, among the outcomes that actually can occur, each sum counts exactly the same results. They just use different ways of identifying those results.

$\endgroup$
3
  • $\begingroup$ Thank you, this is a fantastic answer, that goes into great detail about all sources of uncertainty. The elaboration on the single versus double summation in particular is much appreciated. I've seen both formulas being used to define covariance, without an explanation of why there was a difference. So, if I understand correctly, the single summation is used when a sequence of items from the probability space are specified, which is the case when we formulate the data as an N-dimensional vector of discrete random variables? $\endgroup$
    – saeranv
    Commented Jan 14, 2021 at 9:28
  • 1
    $\begingroup$ Re: the lecture notes omitting the 1/N factor, I'll also add that the nots also omit that there's a 1/N factor in the equivalence between the product of the standard deviation of X and Y, and the magnitude of the X and Y vectors. That may be why it's omitted, since the Cov(X, Y) is divided by the sd(X) and sd(Y) which cancels out the two 1/N factors. So Cov(X,Y) / sd(X) sd(Y) = (dot(X Y)/N) / ((|X||Y|)/N) = dot(X Y) / |X||Y| $\endgroup$
    – saeranv
    Commented Jan 14, 2021 at 9:30
  • $\begingroup$ Good point about the cancellation of factors of $n$. One could also simply multiply by $1/n$ in the definition of the "dot product" used for this exercise--it would not be the usual inner product on $\mathbb R^n$, but it would be an inner product, so calling it "the dot product" is only a little bit of a stretch. $\endgroup$
    – David K
    Commented Jan 14, 2021 at 13:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .