Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It only takes a minute to sign up.
Sign up to join this community
As far as I know orthogonality is a linear algebraic concept, where for a 2D or 3D case if the vectors are perpendicular we say they are orthogonal. Even it is OK for higher dimensions. But when it comes to random variables I cannot figure out orthogonality. I saw that somewhere if the expectation of 2 random variables $X$ and $Y$ is zero ( $E[XY] = 0$ ) then the random variables are orthogonal. How is that possible?
Is orthogonality in linear algebra and probability and statistics same?
Orthogonality comes from the idea of vanishing inner product. In case of random variables $$ \mathbb E \left [ X\right ] = \int_{-\infty}^\infty xd\mu_X $$ so, orthogonal RVs are those with $$ \mathbb E \left [ XY\right ] = \int_{-\infty}^\infty \int_{-\infty}^\infty xy d\mu_X d\mu_Y = 0 $$
Orthogonal means the vectors are at perpendicular to each other. We state that by saying that vectors x and y are orthogonal if their dot product (aka inner product) is zero, i.e. $x^\intercal y$=0.
However for vectors with random components, the orthogonality condition is modified to be Expected Value$E[x^\intercal y]=0$. This can be viewed as saying that for orthogonality, each random outcome of $x^\intercal y$ may not be zero, sometimes positive, sometimes negative, possibly also zero, but Expected Value $E[x^\intercal y]=0$. Keeping in mind, expected value is the same thing as the mean or average of possible outcomes.
Naturally when talking about orthogonality, we are talking about vectors.
The random variables $X$ and $Y$ can be thought of as vectors in a vector space (of infinite dimensions), equipped with an inner product, namely $\langle X, Y \rangle = \mathbb{E}[XY]$. This inner product defines a norm; for a random variable $X$, the norm is $\sqrt{\mathbb{E}[X^2]}$.
The inner product also satisfies symmetry, linearity and positivity as required of any inner product. We call two random variables $X$ and $Y$ as orthogonal when $\langle X, Y \rangle = \mathbb{E}[XY] = 0$.
If $\langle X, Y \rangle$ = 0, then we say $X$ and $Y$ are orthogonal, where $X, Y$ are vectors in an inner product space with inner product $\langle \cdot, \cdot \rangle$.
Now, let $X, Y$ denote two random variables. Suppose $\langle X, Y \rangle = Cov(X,Y),$ where the latter denotes the covariance of $X$ and $Y.$ Then, one can verify that this is indeed an inner product (check the four properties of an inner product).
But, we also know that $Cov(X,Y) = \mathbb{E} [XY] - \mathbb{E} [X]\mathbb{E} [Y],$ so we have that $$\langle X, Y \rangle = \mathbb{E} [XY] - \mathbb{E} [X]\mathbb{E} [Y].$$ If $X$ and $Y$ are independent, as the term is used in probability theory, then $\mathbb{E} [XY] = \mathbb{E} [X]\mathbb{E} [Y],$ so $$\langle X, Y \rangle = \mathbb{E} [XY] - \mathbb{E} [X]\mathbb{E} [Y] = \mathbb{E} [X]\mathbb{E} [Y] - \mathbb{E} [X]\mathbb{E} [Y] = 0.$$ Therefore, $X$ and $Y$ are orthogonal, as the term is used in linear algebra.
