6
$\begingroup$

I am currently working my way through Poole's Linear Algebra, 4th Edition, and I am hitting a bit of a wall in regards to a particular example in the chapter on least squares solutions. The line $y=a+bx$ that "best fits" the data points $(1,2)$, $(2,2)$, and $(3,4)$ can be related to the (inconsistent) system of linear equations $$a+b=2$$ $$a+2b=2$$ $$a+3b=4$$ with matrix representation $$A\mathbf{x}=\begin{bmatrix}1&1\\1&2\\1&3\\\end{bmatrix}\begin{bmatrix}a\\b\\\end{bmatrix}=\begin{bmatrix}2\\2\\4\\\end{bmatrix}=\mathbf{b}$$ Using the least squares theorem, Poole shows that the least squares solution of the system is $$\overline{\mathbf{x}}=\left(A^T A \right)^{-1} A^T \mathbf{b}=\left(\begin{bmatrix}3&6\\6&14\\\end{bmatrix}\right)^{-1}\begin{bmatrix}8\\18\\\end{bmatrix}=\begin{bmatrix}\frac{7}{3}&-1\\-1&\frac{1}{2}\\\end{bmatrix}\begin{bmatrix}8\\18\\\end{bmatrix}=\begin{bmatrix} \frac{2}{3}\\1\\\end{bmatrix}$$ so that the desired line has the equation $y=a+bx=\frac{2}{3} +x$. The components of the vector $\overline{\mathbf{x}}$ can also be interpreted as the coefficients of the columns of $A$ in the linear combination of the columns of $A$ that produces the projection of $\mathbf{b}$ onto the column space of $A$ [which the Best Approximation Theorem identifies as the best approximation to $\mathbf{b}$ in the subspace $\mathrm{col}(A)$]. In other words, the projection of $\mathbf{b}$ onto $\mathrm{col}(A)$ can be found from the coefficients of $\overline{\mathbf{x}}$ by $$\mathrm{proj}_{\mathrm{col}(A)}(\mathbf{b})=\frac{2}{3}\begin{bmatrix}1\\1\\1\\\end{bmatrix}+1\begin{bmatrix}1\\2\\3\\\end{bmatrix}=\begin{bmatrix}\frac{5}{3}\\\frac{8}{3}\\\frac{11}{3}\\\end{bmatrix}$$ But when I try to calculate $\mathrm{proj}_{\mathrm{col}(A)}(\mathbf{b})$ directly [taking $\mathbf{a}_{1}$ and $\mathbf{a}_{2}$ to be the first and second columns of $A$, respectively], I get $$\mathrm{proj}_{\mathrm{col}(A)}(\mathbf{b})=\left(\frac{\mathbf{a}_{1}\cdot\mathbf{b}}{\mathbf{a}_{1}\cdot\mathbf{a}_{1}}\right)\mathbf{a}_{1}+\left(\frac{\mathbf{a}_{2}\cdot\mathbf{b}}{\mathbf{a}_{2}\cdot\mathbf{a}_{2}}\right)\mathbf{a}_{2}=\left(\frac{\begin{bmatrix}1\\1\\1\\\end{bmatrix}\cdot\begin{bmatrix}2\\2\\4\\\end{bmatrix}}{\begin{bmatrix}1\\1\\1\\\end{bmatrix}\cdot\begin{bmatrix}1\\1\\1\\\end{bmatrix}}\right)\begin{bmatrix}1\\1\\1\\\end{bmatrix}+\left(\frac{\begin{bmatrix}1\\2\\3\\\end{bmatrix}\cdot\begin{bmatrix}2\\2\\4\\\end{bmatrix}}{\begin{bmatrix}1\\2\\3\\\end{bmatrix}\cdot\begin{bmatrix}1\\2\\3\\\end{bmatrix}}\right)\begin{bmatrix}1\\2\\3\\\end{bmatrix}$$ $$=\frac{8}{3}\begin{bmatrix}1\\1\\1\\\end{bmatrix}+\frac{18}{14}\begin{bmatrix}1\\2\\3\\\end{bmatrix}=\begin{bmatrix}\frac{8}{3}\\\frac{8}{3}\\\frac{8}{3}\\\end{bmatrix}+\begin{bmatrix}\frac{9}{7}\\\frac{18}{7}\\\frac{27}{7}\\\end{bmatrix}=\begin{bmatrix}\frac{83}{21}\\\frac{110}{21}\\\frac{137}{21}\\\end{bmatrix}$$ I am quite confident that my calculation is incorrect, for a number of reasons. For example, when I take the component of $\mathbf{b}$ orthogonal to $\mathrm{col}(A)$ $$\mathrm{perp}_{\mathrm{col}(A)}(\mathbf{b})=\mathbf{b}-\mathrm{proj}_{\mathrm{col}(A)}(\mathbf{b})=\begin{bmatrix}2\\2\\4\\\end{bmatrix}-\begin{bmatrix}\frac{83}{21}\\\frac{110}{21}\\\frac{137}{21}\\\end{bmatrix}=\begin{bmatrix}-\frac{41}{21}\\-\frac{68}{21}\\-\frac{53}{21}\\\end{bmatrix}$$ I get a vector that is not perpendicular to either $\mathbf{a}_{1}$ or $\mathbf{a}_{2}$, indicating that this vector is not in the orthogonal complement of $\mathrm{col}(A)$. Can somebody help me identify where I'm going wrong in my attempt to calculate the projection of $\mathbf{b}$ onto $\mathrm{col}(A)$?

$\endgroup$
0
4
$\begingroup$

The column space of $A$, namely $U$, is the span of the vectors $\mathbf{a_1}:=(1,1,1)$ and $\mathbf{a_2}:=(1,2,3)$ in $\Bbb R ^3$, and for $\mathbf{b}:=(2,2,4)$ you want to calculate the orthogonal projection of $\mathbf{b}$ in $U$; this is done by $$ \operatorname{proj}_U \mathbf{b}=\langle \mathbf{b},\mathbf{e_1} \rangle \mathbf{e_1}+\langle \mathbf{b},\mathbf{e_2} \rangle \mathbf{e_2}\tag1 $$ where $\mathbf{e_1}$ and $\mathbf{e_2}$ is some orthonormal basis of $U$ and $\langle \mathbf{v},\mathbf{w} \rangle:=v_1w_1+v_2w_2+v_3 w_3$ is the Euclidean dot product in $\Bbb R ^3$, for $\mathbf{v}:=(v_1,v_2,v_3)$ and $\mathbf{w}:=(w_1,w_2,w_3)$ any vectors in $\Bbb R ^3$.

Then you only need to find an orthonormal basis of $U$; you can create one from $\mathbf{a_1}$ and $\mathbf{a_2}$ using the Gram-Schmidt procedure, that is $$ \mathbf{e_1}:=\frac{\mathbf{a_1}}{\|\mathbf{a_1}\|}\quad \text{ and }\quad \mathbf{e_2}:=\frac{\mathbf{a_2}-\langle \mathbf{a_2},\mathbf{e_1} \rangle \mathbf{e_1}}{\|\mathbf{a_2}-\langle \mathbf{a_2},\mathbf{e_1} \rangle \mathbf{e_1}\|}\tag2 $$ where $\|{\cdot}\|$ is the Euclidean norm in $\Bbb R ^3$, defined by $\|\mathbf{v}\|:=\sqrt{\langle \mathbf{v},\mathbf{v} \rangle}=\sqrt{v_1^2+v_2^2+v_3^2}$.

Your mistake is that you assumed that $$ \operatorname{proj}_U\mathbf{b}=\frac{\langle \mathbf{b},\mathbf{a_1} \rangle}{\|\mathbf{a_1}\|^2}\mathbf{a_1}+ \frac{\langle \mathbf{b},\mathbf{a_2} \rangle}{\|\mathbf{a_2}\|^2}\mathbf{a_2}\tag3 $$ however this is not true because $\mathbf{a_1}$ and $\mathbf{a_2}$ are not orthogonal.

$\endgroup$
0
$\begingroup$

Aaand I wasn’t using an orthogonal basis for the subspace. The columns of $A$ are linearly independent, which means a least squares solution exists, but they are not orthogonal, which explains why my calculation of the projection of the vector $\mathbf{b}$ onto the column space of $A$ yielded an incorrect result. Applying the Gram-Schmidt Method to the columns of $A$ produces an orthogonal basis for $\mathrm{col}(A)$, which can then be used to calculate the projection.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.