what is the difference between statistically independent and linearly independent concepts?

  • 2
    $\begingroup$ Wikipedia has great articles on these two topics $\endgroup$ – ClassicStyle Sep 2 '14 at 18:25
  • 1
    $\begingroup$ They are unrelated concepts in different fields of mathematics. $\endgroup$ – André Nicolas Sep 2 '14 at 18:27
  • $\begingroup$ @TylerHG, I readed the Wiki aricles , but they go deep in the two concepts , so i get lost without having a clear idea :( $\endgroup$ – Learner Sep 2 '14 at 18:53
  • $\begingroup$ @Learner It's a bit odd to take two concepts whose definitions are totally unrelated to each other and then ask "what's the difference?" If you were thinking that sharing one word in their names was enough to imply some sort of relationship, then you're in for a lot of big surprises while studying mathematics :) $\endgroup$ – rschwieb Sep 3 '14 at 12:48
  • 2
    $\begingroup$ @rschwieb Both the concepts of linear independence and statistical independence are encountered in the world of statistics, often in close proximity (e.g., one frequently sees "covariance is a measure of linear dependence"). Furthermore, linear independence has clear mathematical implications vis-à-vis statistical independence. I therefore don't agree that they are completely unrelated concepts from different branches of maths. The links and differences between them are frequent sources of confusion in statistics, so I think this question is rather worthwhile. $\endgroup$ – davincisghost Sep 3 '14 at 13:46

Consider a simple scenario in which you have two non-zero, non-constant, $n$-dimensional data vectors $\mathbf{X}$ and $\mathbf{Y}$.

They are linearly independent if there is no non-zero scalar $\alpha$ such that

$\alpha \mathbf{X} - \mathbf{Y} = \mathbf{0}$

In other words, there is no non-zero multiplicative constant $\alpha$ that will transform $\mathbf{X}$ into $\mathbf{Y}$. Geometrically, this means that the vectors $\mathbf{X}$ and $\mathbf{Y}$ do not lie on the same line.

The two vectors $\mathbf{X}$ and $\mathbf{Y}$ are statistically independent if and only if their joint probability density is the product of their marginal probability densities, i.e.,

$f(\mathbf{X}, \mathbf{Y}) = f_X(\mathbf{X}) \cdot f_Y(\mathbf{Y})$

This implies

cov$(\mathbf{X}, \mathbf{Y}) = \mathbf{0}$

(though the reverse implication is not true generally).

The two concepts are linked insofar as if the two vectors are not linearly independent then they can also not be statistically independent. For example, if for some non-zero scalar $\alpha$ we have

$\alpha \mathbf{X} = \mathbf{Y}$


cov$(\mathbf{X}, \mathbf{Y}) = \text{cov}(\frac{1}{\alpha}\mathbf{Y}, \mathbf{Y}) = \frac{1}{\alpha} \text{var} (\mathbf{Y}) \ne \mathbf{0}$

However, linear independence of $\mathbf{X}$ and $\mathbf{Y}$ does not guarantee statistical independence (it is possible to have cov$(\mathbf{X}, \mathbf{Y}) \ne \mathbf{0}$ even if $\mathbf{X}$ and $\mathbf{Y}$ are linearly independent).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.