1
$\begingroup$

Introduction:

This lecture explains how can linear algebraic methods be used to solve linear regression problems.

From my understanding, If there is no solution to $Ax=b$, we can project $b$ into the space and get $x$ which will be the closest solution. Although $A$ must be orthogonal for the error $e = b - Ax$ (or it should be in the null space of $A^T$, which is equivalent to orthogonality) in the approximation since then the inner product of $A$ and $e$ would give us the equation.

If we are working in the spaces of 3 dimensions or higher (or planes of 2 dimensions and higher), then our projection could be represented as:

$P = A(A^TA)^{-1}A^T$

Problem:

At this point of the lecture, Professor Strang presents linear function $y=C+Dt$ and its equation for each point in the graph, then proceeds to show the problem which is:

$Ax = b$

Somehow according to the graph, variables are substituted with following vectors:

$\begin{bmatrix}1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}\begin{bmatrix}C \\D\end{bmatrix} = \begin{bmatrix}1 \\ 2 \\ 3 \end{bmatrix}$

I don't understand representation of the matrix $A$, why is first column padded with ones? Is it specifically padded so it can work in projection equation above which expects $A$ to be in matrix form? If not then what does it represent?

Question:

Shortly, what is the proper way to use this projection matrix equation? ($P = A(A^TA)^{-1}A^T$)

Thank you.

$\endgroup$
1
$\begingroup$

If you multiply the matrix and the vector $\begin{bmatrix} C\\D\end{bmatrix}$, you see, that the lines read \begin{align}C+D\cdot 1&=1 \\ C+D \cdot 2 &= 2 \\ C+D \cdot 3 &= 3. \end{align}

At this point, it should be pretty clear, where the $1$ in the frst collumn does come from.

In essence, you search for a linear combination of basis functions. The linear coefficients, here $C$ and $D$, are represented in the vector of unknowns. The Basis functions, that you try to combine, in this case the constant function $1$ and the linear function $t$, are in the matrix. Each collumn contains the evaluations of these basis function on the point, you have the data (in our case here the evaluations points are $t=1,2,3$). The constant function is always $1$, that's where the constants are coming from.

$\endgroup$
2
  • $\begingroup$ Thank you! I should've tried matrix multiplication first. Also if I may ask, from what I understood, first column also represents derivative correct? Which would make sense since slope of the tangent line in most of the linear functions is obviously one. $\endgroup$ – ShellRox Apr 17 '18 at 20:15
  • 1
    $\begingroup$ It has nothing to do with the slope! The collumns of the matrix are just the evaluations of the basis functions! You can see this, by trying to fit a combination of $x^2$ and $e^{2x}$ to your data. $\endgroup$ – Laray Apr 18 '18 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.