I have been self-studying statistics recently, and the apparent similarities between linear algebra (especially Hilbert spaces) and statistics have been popping out to me. Linear independence gets replaced with statistical independence. The mean is like an origin to which random variables can be translated to change from an affine space into a vector space. The covariance is basically just an inner product.

At first I thought that these similarities might be coincidental, but after a lot of research, it became clear that they are not. There appears to be quite a lot of literature on this topic, but I have not yet studied this relationship very closely and am in the process of finding a textbook to read.

That said, based on the research that I have done so far, there is still one connection to vectors that has not been discussed anywhere I've looked: the Chi-Squared distribution.

Given the analogies described above, it seems obvious that the analogy for independent, standard normal random variables of the form

$$X=\frac{x-\mu_x}{\sigma_x}, \ \ Y=\frac{x-\mu_y}{\sigma_y}, \text{ etc.} $$

would be orthogonal unit vectors.

With this understanding, one might construct a "random variable vector" ( $\chi$ ) of the form

$${\large \chi}(a,b,c) = aX+bY+cZ= \pmatrix{a \\ b \\ c} $$

where $a$, $b$, and $c$ are constant, real-valued, and non-random. This vector is itself a normally distributed random variable with mean $\mu_\chi=a\mu_x+b\mu_y+c\mu_z=0$ and variance $\sigma_\chi^2=a^2\sigma_x^2+b^2\sigma_y^2+c^2\sigma_z^2=a^2+b^2+c^2$. So far, it is obeying all of the analogical rules established above.

That would lead naturally to the Chi-squared distribution

$${\large \chi^2} =\pmatrix{1 & 1 & 1} \pmatrix{1 \\ 1 \\ 1}=X^2+Y^2+Z^2 $$

which is not a "vector" (as expected). However, it is not clear to me how to interpret its meaning - especially its mean (in this case, $\mu_{\chi^2}=3$) or variance (in this case, $\sigma_{\chi^2}^2=6$).

So I have 3 inter-related questions:

1) Does it make sense to interpret independent, standard normal random variables as being akin to "orthogonal unit vectors"?

2) If it does, is there any meaning or utility in constructing a "random variable vector" as described above? If there is not, is there another way to construct a random variable vector that is meaningful?

3) Finally, is there any sense in which the Chi-squared distribution can be seen as an inner product between a "random variable vector" and itself? Can that inner product be generalized to be between different vectors? Can it be used to construct a "norm" on this space (say, with the Chi distribution)? If so, what exactly does that mean?

Note: I know that this is a question from ignorance built on possibly tenuous analogies. Since I am self-studying, I can't help but draw connections between this new material and the sort of math that I am already comfortable with. If there is any way that I can improve this question, please let me know.


Interesting question!

First of all, yes, it does make sense to define independent random variables as different components of a vector. For example, the position of a particle in three dimensions might well be determined by $x,y,$ and $z$ each being a standard normal random variable. So it can make literal geometric sense to do as you suggest.

People make random vectors all the time. They see common use in linear regression with multiple variables, for example. Observations of several variables at once are often listed forming a vector. Many of the regression formulas are written with matrices and vectors.

You can call something a vector if the collection of things like it satisfies the rules of a vector space. Identity, inverse, commutativity of addition, multiplication by a constant, and so forth. I see no reason why you couldn't make your vectors of normal random variables into a vector space. However, you need to decide if you want to use just normal random variables, or allow any random variables. The sum of two normal random variables is normal, but that isn't true for many other distributions. Also you have to be careful what you are using as scalars, if you want to multiply components by them!

The problem comes in when you sum them. If you add three random variables together with coefficients, you get, as you pointed out, one random variable. The closest geometric analogy would be using the Manhattan metric in three dimensions, where your sum might be the distance between two points moving only along a rectangular grid. But three components of a vector do not sum up to one of anything in particular, generally.

The $\chi^2$ random variable for positive integer degrees of freedom can be considered as a sum of squared normal variables, so it does look rather like a dot product, doesn't it? However, as I pointed out above, you need to know what your scalars are. If they are regular numbers, then a chi-square random variable is not a number and so cannot be a dot product result. If you wanted to use chi square random variables, then you have the problem of multiplying a $Z$ by a $\chi^2$ and not getting a $Z$ back again, so that you wouldn't have a vector any more.

If you were to make a very general space, where components can be any random variable at all, and the scalars are also random variables, then you might be able to make it work. Covariance might not makes sense any more as a dot product in that case, however.


It might be helpful to think of our addition across all df in the first place: if we had to establish if they were dependent/independent, we would certainly need to evaluate an inner product, but in the design of this test, we impose this 'orthogonality' on our degrees of freedom as a feature, not a bug, since it just makes the chi-squared test that much more conservative in determining dependence.

With respect to the inner product, I would go backwards: start with a l2 norm, which we're using, and then use the induced inner product from that norm. It seems that our inner product space is that induced by l2 norm on the vector space of errors residuals, not the original data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.