2
$\begingroup$

Given data points in the form $(x,y,f(x,y)) = (1,1,7),(1,3,0),(1,-1,8)$, find the least squares solution $\hat x$ to the system of equations $Ax=b$.

Is there enough information to use least squares? The solution created $A$ using the $x$ and $y$ coordinates and $b$ using $f(x,y)$. I do not understand this problem. How can you solve for $\hat x$ when you do not know anything about the function? Isn't it presumptuous to assume a linear relationship?

$\endgroup$
1
$\begingroup$
  1. Yes, you got enough information to construct the projection matrix $H$, $$ H=X(X'X)^{-1}X', $$ where $ X=\begin{pmatrix} 1,\,\, 1\\ 1,\,\, 3\\ 1,-1 \end{pmatrix} $, then $Hy=\hat{y}$ where $y=(7,0,8)'$, gives you the least square solution which is $(5,1,9)'$.

  2. You do not necessarily assume linear relationship between $y$ and $x$, you may view the least square solution as a linear approximation of $f(y,x)$ based on $x$. So, basically your assumption is of more technical nature like estimability of the linear (affine) approximation of $y$ using $x$.

$\endgroup$
0
$\begingroup$

We are given a sequence of measurements $\left\{ x_{k}, y_{k}, f_{k} \right\}_{k=1}^{3}$. Generally, we could hope to find three fit parameters, a trial function like $$ f(x,y) = c_{0}g_{0}(x,y) + c_{1}g_{1}(x,y) + c_{2}g_{2}(x,y) $$ where the functions $g$ are linearly independent.

You asked about the linear case where $$ f(x,y) = c_{0} + c_{1}x + c_{2}y. $$ That problem looks like this $$ \begin{align} \mathbf{A} c &= F \\[5pt] % \left[ \begin{array}{ccc} 1 & x_{1} & y_{1} \\ 1 & x_{2} & y_{2} \\ 1 & x_{3} & y_{3} \end{array} \right] \left[ \begin{array}{c} c_{0} \\ c_{1} \\ c_{2} \end{array} \right] &= \left[ \begin{array}{c} f_{1} \\ f_{2} \\ f_{3} \end{array} \right] \\[5pt] % \left[ \begin{array}{ccr} 1 & 1 & 1 \\ 1 & 1 & 3 \\ 1 & 1 & -1 \end{array} \right] \left[ \begin{array}{c} c_{0} \\ c_{1} \\ c_{2} \end{array} \right] &= \left[ \begin{array}{c} 7 \\ 0 \\ 7 \end{array} \right] \end{align} $$ The is no solution vector $c$ which satisfies this equation, so instead of asking that $\mathbf{A}c-F = 0$, we ask instead that $\mathbf{A}c-F$ be as small as possible.To measure length, we must select a norm, and the choice here is the $2-$norm of least squares.

The least squares solution is $$ c_{LS} = \mathbf{A}^{\dagger}F + \left(\mathbf{I}_{3} - \mathbf{A}^{\dagger}\mathbf{A} \right) y, \quad y\in\mathbb{C}^{3} $$

There is a problem with this model in that the first two column vectors of $\mathbf{A}$ are identical. This matrix has rank $\rho = 2.$ We can't use the normal equations.

Using the SVD, $$ \mathbf{A}^{\dagger} = \frac{1}{24} \left[ \begin{array}{ccr} 4 & 1 & 7 \\ 4 & 1 & 7 \\ 0 & 6 & -6 \\ \end{array} \right] $$ and the full solution is $$ c_{LS} = \frac{1}{24} \left[ \begin{array}{r} 77 \\ 77 \\ -42 \end{array} \right] + \alpha \left[ \begin{array}{r} -1 \\ 1 \\ 0 \end{array} \right]. $$

The least squares fit is the best fit, but it may not be a good fit. That is, given the trial function, the method will find the best parameters. But the trial function may be bad.

The residual error vector $$ r(c_{LS}) = \frac{1}{6} \left[ \begin{array}{r} -14 \\ 7 \\ 7 \end{array} \right]. $$ The total error is $r\cdot r = \frac{49}{6} \approx 8.2.$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.