# Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

1,357 questions
Filter by
Sorted by
Tagged with
24k views

### Why does SVD provide the least squares and least norm solution to $A x = b$?

I am studying the Singular Value Decomposition and its properties. It is widely used in order to solve equations of the form $Ax=b$. I have seen the following: When we have the equation system $Ax=b$, ...
17k views

### Derivative of squared Frobenius norm of a matrix

In linear regression, the loss function is expressed as $$\frac1N \left\|XW-Y\right\|_{\text{F}}^2$$ where $X, W, Y$ are matrices. Taking derivative w.r.t $W$ yields $$\frac 2N \, X^T(XW-Y)$$ Why ...
4k views

### How does the SVD solve the least squares problem?

How do I prove that the least-squares solution for $$\text{minimize} \quad \|Ax-b\|_2$$ is $A^{+} b$, where $A^{+}$ is the pseudoinverse of $A$?
7k views

### Matrix Calculus in Least-Square method

In the proof of matrix solution of Least Square Method, I see some matrix calculus, which I have no clue. Can anyone explain to me or recommend me a good link to study this sort of matrix calculus? ...
20k views

### Difference between least squares and minimum norm solution

Consider a linear system of equations $Ax = b$. If the system is overdetermined, the least squares (approximate) solution minimizes $||b - Ax||^2$. Some source sources also mention $||b - Ax||$. If ...
950 views

1k views

### When is Block-Partitioned Matrix Invertible?

Suppose I have a block partitioned matrix \begin{equation} \begin{bmatrix} \mathbf{X}_1^{\top}\mathbf{X}_1 & \mathbf{X}_1^{\top}\mathbf{X}_2 \\ \mathbf{X}_2^{\top}\mathbf{X}_1 & \mathbf{X}_2^{\...
5k views

### Show that the least squares line must pass through the center of mass

My problem: The point $(\bar x, \bar y)$ is the center of mass for the collection of points in Exercise 7. Show that the least squares line must pass through the center of mass. [Hint: Use a change ...
973 views

3k views

### How to perform a monotonic function fitting of data points?

I'm seeking suggestions for general purpose function fitting of a set of data points, where, based on physical intuition, the relationship is expected to be "monotonic", i.e. the function should be ...
2k views

2k views

### What forms does the Moore-Penrose inverse take under systems with full rank, full column rank, and full row rank?

The normal form $(A'A)x = A'b$ gives a solution to the least square problem. When $A$ has full rank $x = (A'A)^{-1}A'b$ is the least square solution. How can we show that the moore-penrose solves ...
6k views

### Solve an overdetermined system of linear equations

I have doubt to solve this system of equations \begin{cases} x+y=r_1\\ x+z=c_1\\ x+w=d_1\\ y+z=d_2\\ y+w=c_2\\ z+w=r_2 \end{cases} Is it an overdetermined system because I see there are more ...
370 views

### How to Solve Linear Least Squares with Matrix Inequality Constraint

I need to solve the following inequality-constrained least-squares problem in vector $x$ $$\min_{Ax \geq 0} \frac{1}{2} \|Ax-b\|_2^2$$ where matrix $A$ and vector $b$ are given. I am totally ...
### Explain why $x^+=A^+b$ is the shortest possible solution to $A^TA\hat{x}=A^Tb$
I'm was going through the chapter on pseudoinverse in intro to linear algebra by Strang, and it says The vector $x^+=A^+b$ is the shortest possible solution to $A^TA\hat{x}=A^Tb$ Reason: The ...
Find the parabola $At^2+Bt+C$ that best approximates the data set $t= -1,0,1,2,3$ and $b(t) = 5,2,1,2,5$. Would I be using least squares such that $x = (A^TA)^{-1}A^Tb$? Thank you in advance.