# Relationship between projection of $y$ onto $x_1, x_2$ individually vs. projection on both?

This is essentially similar to the question I just asked on cross validated, but here I am going to pose it in a linear algebra way.

Consider $$y \in \mathbb{R}^n$$ and $$x_1, x_2, 1_n \in \mathbb{R}^{n}$$. Suppose you orthogonally project $$y$$ onto $$x_1, 1_n$$ and find the projection of $$y$$ onto the subspace spanned by $$x_1, 1_n$$ can be written as $$\hat{y}_1 = \hat{\beta}_1 x_1 + b_1$$, i.e., a linear combination of $$x_1$$ plus some offset. Now do the same for orthogonal projection of $$y$$ onto $$x_2, 1_n$$ and find $$\hat{y}_2 = \hat{\beta}_2 x_2 + b_2$$.

Now consider projecting $$y$$ onto the subspace spanned by both $$x_1, x_2, 1_n$$ and find $$\hat{y}_{12} = \hat{\gamma}_1 x_1 + \hat{\gamma}_2x_2 + b_{12}$$.

If $$x_1 \perp x_2$$, then I know $$\hat{\beta}_i = \hat{\gamma}_i$$. But what if they're not orthogonal?

What can I say about the relationship between $$\hat{\beta}$$ and $$\hat{\gamma}$$ in this case?

Some specific questions that I am also interested in is if $$\hat{\beta} >0$$, does this imply $$\hat{\gamma} > 0$$? If $$x_1, x_2$$ are linearly dependent, then I don't think this won't be true for one of the coefficients.

• @BrianMoehring I brainfarted when posting this question. So $x_1, x_2, y$ are all $\in \mathbb{R}^n$. Apologies for the confusion. – roulette01 Aug 15 '20 at 17:53
• The subspace spanned by $x_1$ is unidimensional, right? What does this $b_1$ correspond to? – md5 Aug 15 '20 at 21:01
• @md5 If I understand correctly, the first answer should be "yes." $x_1, x_2$ are just $n \times 1$ vectors. The $b$'s are just offset terms. Actually technically, for each individual case, the subspace is spanned by $x_1$ and also $1_n$, a vector of 1s. – roulette01 Aug 15 '20 at 21:03

I cannot say I completely understood what those constants $$b_1$$, $$b_2$$ or $$b_{12}$$ are for. But I understood the gist of your question and I'll try my best.

Say the orthogonal projection of $$y$$ onto the subspace spanned by $$x_1$$ can be written as $$\hat{y}_1 = \hat{\beta}_1 x_1$$, i.e., a linear combination of $$x_1$$. Now we do the same for orthogonal projection of $$y$$ onto $$x_2$$ and find $$\hat{y}_2 = \hat{\beta}_2 x_2$$.

Also we have the projection of $$y$$ onto the subspace spanned by both $$x_1, x_2$$ and find $$\hat{y}_{12} = \hat{\gamma}_1 x_1 + \hat{\gamma}_2x_2$$.

Without loss of generality we can say the vectors $$x_1$$ and $$x_2$$ are unit vectors and represent them by $$\hat{x_1}$$ and $$\hat{x_2}$$. If you don't want to do this, then rewrite all the vectors in terms in terms of $$\hat{x_1}$$ and $$\hat{x_2}$$. So for example, $$\hat{\beta_1}$$ will become $$\hat{\beta_1} ||x_1||$$

Now, consider this statement. The orthogonal projection of $$\hat{y_{12}}$$ onto $$x_1$$ would be the same as $$\hat{y_1}$$ and the orthogonal projection of $$\hat{y_{12}}$$ onto $$x_2$$ would be the same as $$\hat{y_2}$$.

So, by the definition of projection,

$$\hat{y_{12}}.\hat{x_1} = ||\hat{y_1}||$$

$$\implies (\hat{\gamma}_1 \hat{x_1} + \hat{\gamma}_2 \hat{x_2}).\hat{x_1} = ||\hat{y_1}||$$

$$\implies \hat{\gamma}_1 \hat{x_1}.\hat{x_1} + \hat{\gamma}_2 \hat{x_2}.\hat{x_1} = \hat{\beta}_1$$

$$\implies \hat{\gamma}_1 + \hat{\gamma}_2 \hat{x_2}.\hat{x_1} = \hat{\beta}_1 \tag{1}$$

Similarly we can solve $$\hat{y_{12}}.\hat{x_1} = ||\hat{y_2}||$$ to get

$$\implies \hat{\gamma}_1 \hat{x_1}.\hat{x_2} + \hat{\gamma}_2 = \hat{\beta}_2 \tag{2}$$

There you go. We have 2 equations and 2 unknowns.

Obviously we should know the value of $$\hat{x_1}.\hat{x_2}$$, in other words the cosine of the angle between them, to get the required relations. In the case where $$\hat{x_1}$$ and $$\hat{x_2}$$ are orthogonal, $$cos \frac{\pi}{2}=0$$ and hence the result you gave $$\hat{\beta}_i = \hat{\gamma}_i$$.

• Any criticism or follow up to the answer is welcome. – Chaitanya Chavali Mar 12 at 18:06