1
$\begingroup$

This is essentially similar to the question I just asked on cross validated, but here I am going to pose it in a linear algebra way.

Consider $y \in \mathbb{R}^n$ and $x_1, x_2, 1_n \in \mathbb{R}^{n}$. Suppose you orthogonally project $y$ onto $x_1, 1_n$ and find the projection of $y$ onto the subspace spanned by $x_1, 1_n$ can be written as $\hat{y}_1 = \hat{\beta}_1 x_1 + b_1$, i.e., a linear combination of $x_1$ plus some offset. Now do the same for orthogonal projection of $y$ onto $x_2, 1_n$ and find $\hat{y}_2 = \hat{\beta}_2 x_2 + b_2$.

Now consider projecting $y$ onto the subspace spanned by both $x_1, x_2, 1_n$ and find $\hat{y}_{12} = \hat{\gamma}_1 x_1 + \hat{\gamma}_2x_2 + b_{12}$.

If $x_1 \perp x_2$, then I know $\hat{\beta}_i = \hat{\gamma}_i$. But what if they're not orthogonal?

What can I say about the relationship between $\hat{\beta}$ and $\hat{\gamma}$ in this case?

Some specific questions that I am also interested in is if $\hat{\beta} >0 $, does this imply $\hat{\gamma} > 0$? If $x_1, x_2$ are linearly dependent, then I don't think this won't be true for one of the coefficients.

$\endgroup$
3
  • 1
    $\begingroup$ @BrianMoehring I brainfarted when posting this question. So $x_1, x_2, y$ are all $\in \mathbb{R}^n$. Apologies for the confusion. $\endgroup$ – roulette01 Aug 15 '20 at 17:53
  • 1
    $\begingroup$ The subspace spanned by $x_1$ is unidimensional, right? What does this $b_1$ correspond to? $\endgroup$ – md5 Aug 15 '20 at 21:01
  • $\begingroup$ @md5 If I understand correctly, the first answer should be "yes." $x_1, x_2$ are just $n \times 1 $ vectors. The $b$'s are just offset terms. Actually technically, for each individual case, the subspace is spanned by $x_1$ and also $1_n$, a vector of 1s. $\endgroup$ – roulette01 Aug 15 '20 at 21:03
0
$\begingroup$

I cannot say I completely understood what those constants $b_1$, $b_2$ or $b_{12}$ are for. But I understood the gist of your question and I'll try my best.

Say the orthogonal projection of $y$ onto the subspace spanned by $x_1$ can be written as $\hat{y}_1 = \hat{\beta}_1 x_1$, i.e., a linear combination of $x_1$. Now we do the same for orthogonal projection of $y$ onto $x_2$ and find $\hat{y}_2 = \hat{\beta}_2 x_2$.

Also we have the projection of $y$ onto the subspace spanned by both $x_1, x_2$ and find $\hat{y}_{12} = \hat{\gamma}_1 x_1 + \hat{\gamma}_2x_2$.

Without loss of generality we can say the vectors $x_1$ and $x_2$ are unit vectors and represent them by $\hat{x_1}$ and $\hat{x_2}$. If you don't want to do this, then rewrite all the vectors in terms in terms of $\hat{x_1}$ and $\hat{x_2}$. So for example, $\hat{\beta_1}$ will become $\hat{\beta_1} ||x_1||$

Now, consider this statement. The orthogonal projection of $\hat{y_{12}}$ onto $x_1$ would be the same as $\hat{y_1}$ and the orthogonal projection of $\hat{y_{12}}$ onto $x_2$ would be the same as $\hat{y_2}$.

So, by the definition of projection,

$$ \hat{y_{12}}.\hat{x_1} = ||\hat{y_1}|| $$

$$ \implies (\hat{\gamma}_1 \hat{x_1} + \hat{\gamma}_2 \hat{x_2}).\hat{x_1} = ||\hat{y_1}||$$

$$ \implies \hat{\gamma}_1 \hat{x_1}.\hat{x_1} + \hat{\gamma}_2 \hat{x_2}.\hat{x_1} = \hat{\beta}_1$$

$$ \implies \hat{\gamma}_1 + \hat{\gamma}_2 \hat{x_2}.\hat{x_1} = \hat{\beta}_1 \tag{1}$$

Similarly we can solve $ \hat{y_{12}}.\hat{x_1} = ||\hat{y_2}|| $ to get

$$ \implies \hat{\gamma}_1 \hat{x_1}.\hat{x_2} + \hat{\gamma}_2 = \hat{\beta}_2 \tag{2}$$

There you go. We have 2 equations and 2 unknowns.

Obviously we should know the value of $\hat{x_1}.\hat{x_2}$, in other words the cosine of the angle between them, to get the required relations. In the case where $\hat{x_1}$ and $\hat{x_2}$ are orthogonal, $cos \frac{\pi}{2}=0$ and hence the result you gave $\hat{\beta}_i = \hat{\gamma}_i$.

$\endgroup$
1
  • $\begingroup$ Any criticism or follow up to the answer is welcome. $\endgroup$ – Chaitanya Chavali Mar 12 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.