# Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

137 questions
Filter by
Sorted by
Tagged with
4k views

### How does the SVD solve the least squares problem?

How do I prove that the least-squares solution for $$\text{minimize} \quad \|Ax-b\|_2$$ is $A^{+} b$, where $A^{+}$ is the pseudoinverse of $A$?
2k views

### What forms does the Moore-Penrose inverse take under systems with full rank, full column rank, and full row rank?

The normal form $(A'A)x = A'b$ gives a solution to the least square problem. When $A$ has full rank $x = (A'A)^{-1}A'b$ is the least square solution. How can we show that the moore-penrose solves ...
3k views

### Least squares Problem with Non Negativity Constraints

Let $\mathbf{x}=[x_1,\ldots,x_K]$. I have the following optimization problem: \begin{array}{rl} \min \limits_{\mathbf{x}} & \| \mathbf{Ax}-\mathbf{b} \|^2 \\ \mbox{s.t.} & x_k\ge 0, \forall ...
1k views

### Closed Form Solution of $\arg \min_{x} {\left\| x - y \right\|}_{2}^{2} + \lambda {\left\|x \right\|}_{2}$ - Tikhonov Regularized Least Squares

The problem is given by: $$\arg \min_{x} \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} + \lambda {\left\|x \right\|}_{2}$$ Where $y$ and $x$ are vectors. $\|\cdot\|_2$ is Euclidean norm. In the ...
847 views

### Singular value decomposition proof

I need help in the following question. I'm not sure how to even begin to answer this. What is a possible proof for the following question? If $A$ is an $m \times n$ matrix and $b$ is an $m$-vector, ...
8k views

### Difference between orthogonal projection and least squares solution

When you find the least squares solution you solve $$A^TA = A\vec b$$ but to find the orthogonal projection into the "subspace" A, you multiply this result (the least squares solution) with the ...
17k views

### Derivative of squared Frobenius norm of a matrix

In linear regression, the loss function is expressed as $$\frac1N \left\|XW-Y\right\|_{\text{F}}^2$$ where $X, W, Y$ are matrices. Taking derivative w.r.t $W$ yields $$\frac 2N \, X^T(XW-Y)$$ Why ...
3k views

### Orthogonal Projection of $z$ onto the Affine set $\left\{ x \mid A x = b \right\}$

Suppose $A$ is fat(number of columns > number of rows) and full row rank. The projection of $z$ onto $\{x\mid Ax = b\}$ is (affine) $$P(z) = z - A^T(AA^T)^{-1}(Az-b)$$ How to show this? ...
10k views

### Unique least square solutions

There is a theorem in my book that states: If $A$ is $m\times n$, then the equation $Ax = b$ has a unique least square solution for each $b$ in $\mathbb{R}^m$. But can we find a counter-example to ...
24k views

### Why does SVD provide the least squares and least norm solution to $A x = b$?

I am studying the Singular Value Decomposition and its properties. It is widely used in order to solve equations of the form $Ax=b$. I have seen the following: When we have the equation system $Ax=b$, ...
20k views

### Difference between least squares and minimum norm solution

Consider a linear system of equations $Ax = b$. If the system is overdetermined, the least squares (approximate) solution minimizes $||b - Ax||^2$. Some source sources also mention $||b - Ax||$. If ...
7k views

### Matrix Calculus in Least-Square method

In the proof of matrix solution of Least Square Method, I see some matrix calculus, which I have no clue. Can anyone explain to me or recommend me a good link to study this sort of matrix calculus? ...
6k views

### Least Square Approximation for Exponential Functions

I'm a little confused on how to approach the following problem. Use the least square method to fit a curve of the form $y=a\cdot b^x$ to a collection of $n$ data points $(x_1,y_1),...,(x_n,y_n)$ ...
842 views

### Is the unique least norm solution to $Ax=b$ the orthogonal projection of b onto $R(A)$?

True or false: The unique least norm solution to $Ax=b$ is the orthogonal projection of b onto $R(A)$ So, first isn't this the definition of least squares? The only incorrect thing I can think of is ...
169 views

### Solve Linear Least Squares Problem with Unit Simplex Constraint

$$\min_x ||Ax - b||_2\; \;\text{given }x \geq 0\;\;\text{and}\;\;\textbf{1}^Tx = 1$$ I am trying to do the above optimization, I was using common Quadratic programming libraries but their speed is ...
950 views

5k views

### Show that the least squares line must pass through the center of mass

My problem: The point $(\bar x, \bar y)$ is the center of mass for the collection of points in Exercise 7. Show that the least squares line must pass through the center of mass. [Hint: Use a change ...
2k views

5k views

### How come least square can have many solutions?

I know there always exists a least-square solution $\hat{x}$, regardless of the properties of the matrix $A$. However, I keep finding online that least-square can have infinitely many solutions, if $A$...
204 views

### Best fitting in a curve of the form $Ax^B+C$

I am trying to fit data of the form $(x_i,y_i)$, $i=1,\ldots,n$, in a curve of the form $y=Ax^B+C$, where $B\in (0,1)$. All three constants $A,B,C$ are to be determined optimally (no particular norm ...
842 views

### Finding the null space of a matrix by least squares optimization?

I've had some idea I could do something like this: $${\bf v_o}=\min_{\bf v} \{\|{\bf M(v+r)}\|_2^2 + \epsilon\|{\bf v}\|_2^2\}$$ for a random vector $\bf r$ and then $\bf v$ should point in the ...
96 views

### Least Square fit for signal data (360 points)

I would like to analyze data to get the maximum value out of 360 points. I used least square fitting because I get the data from signal strengths. I want to remove any outliers I get from my data ...
1k views

### Solving least squares problem using partial derivatives

Let's say we want to solve a linear regression problem by choosing the best slope and bias with the least squared errors. As example, let the points be $x=[1, 2, 3]$ and $y=[1,2,2]$. This quadratic ...
2k views

255 views

### How does minimum squared error relate to a linear system?

Given some system $U*x = b$, I've solved for $x^*$, the least squares solution. I then compute the minimum squared error by $||U*x^* - b||^2$. I know that the least squares solution minimizes the ...
151 views

### Least squares problem: find the line through the origin in $\mathbb{R}^{3}$

The problem is as follows: "Please set up (but do not solve) the normal equations for the following least squares approximation problem: Find $(a, b, c, d)$ such that the plane H described by \$ax + ...
How to minimize the follow optimization $$\begin{array}{ll} \text{minimize} & \| \mathbf{A}\mathbf{x} - \mathbf{b} \|^2\\ \text{subject to} & {\mathbf{x}}^{T} \mathbb{1} = \mathbb{1}\\ & \... 2answers 516 views ### Solver for Norm Constraint Least Squares I'm looking for a numerical solution to the constrained least squares problem below:$$ \min_\mathbf{x}\|\mathbf{a+Bx}\|^2 ~~\text{s.t}~~\|\mathbf{x}\|^2 \leq \alpha^2$$where \mathbf{a} \in \mathbb{... 1answer 118 views ### How to Solve Linear Least Squares Problem with Box Constraints [closed] How could one solve the following Least Squares with Box Constraints problem:$$\begin{aligned} \arg \min_{x \in \mathbb{R}^{n}} \quad & \frac{1}{2} {\left\| A x - b \right\|}_{2}^{2} \\ \text{...
The standard form for ridge regression aims to minimize the following cost function. $$\min\ \ \sum_i(y_i-x_i^T\beta)^2 + \lambda\sum_j\beta^2_j$$ As described here, it's possible to differentiate ...