Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere. See https://en.wikipedia.org/wiki/Least_squares

learn more… | top users | synonyms

0
votes
0answers
4 views

Weight Modification for Computationally-Efficient Nonlinear Least Squares Optimization

There was a time where I could figure this out for myself, but my math skills are rustier than I thought, so I have to humbly beg for help. Thank you in advance. I am solving a weighted nonlinear ...
0
votes
1answer
11 views

Least squares without solution

Talking about simple linear regression (k=1), in which cases the Normal Equations have unique solution? And infinite? And when the Normal equations have no solution?
0
votes
0answers
17 views

mean-deviation form, why orthogonal?

This is from my textbook Why the column of the new design matrix are orthogonal? for example, let say $A=\begin{pmatrix} 1& 1& 4\\ 1& 2& 0\\ 1& 3& 2 \end{pmatrix}$ ...
0
votes
0answers
14 views

QR factorization for least squares

This is from my textbook I don't undertand why small errorr in $A^TA$ can lead to large error in cofficient matrix? Because A=QR, so there should be no difference to use A or QR anyway.Could someone ...
0
votes
1answer
12 views

How to expand the matrix in Matlab (forming $A$ matrix in least square problem)

If I have 5 data points (actually, this could be vary huge number): $a = [1;2;3;4;5]$ To each data point, I can form a matrix $[a(i),2a(i)]$. (actually, I have to find the matrix, this matrix needs ...
0
votes
0answers
8 views

Aligning matrices, normalization. Calculating coefficients.

So as a pre-task for my upcoming exam this is one of the rehearsal assignments. I can't wrap my head around this one at all, haven't seen anything like it earlier, and I can't seem to find any ...
0
votes
1answer
29 views

Least Squares Revisited

I am reading a paper on regression and there seems to be a simple substitution but I just cannot get my head around it. My question is how you go from equation (3) and (4) to (5)? Please let me know ...
0
votes
0answers
16 views

Efficient estimator: best choice for the weight matrix in the Weighted Least Squares Estimation

I am facing the linear regression problem in the form: $$y = \Phi\theta+\eta$$ where $y\in\mathbb{R}^N$ is the vector of the measurements (the available data), $\Phi\in\mathbb{R}^{N\times n}$ the ...
1
vote
1answer
61 views

Finding the normal equation

After my semester at Umich my mathematics professor issued me an abundance of problems to keep my head in the game during the summer. One of the questions which threw me off was finding the normal ...
0
votes
1answer
26 views

How to fit a set of 3D points to a helical curve?

suppose I have a set of points in $\mathbb{R}^3$, and I want to find an arbitrary helix which best approximates these points. An arbitrary helix in $\mathbb{R}^3$ can be parametrized as ...
0
votes
0answers
17 views

Finding the approximate confidence interval for a paramter

Intro From the previous exercise, we were instructed to find a parameter $b$ using maximum likelihood and least square methods in the Rayleigh distribution, given by $$ {f_X}(x) = ...
1
vote
1answer
20 views

Asymmetric Least Squares Conversion from Equation to Matrix

In solving for asymmetric least squares baseline correction as defined in the article by Eilers and Boelens, the general equation is defined as: $$S = \displaystyle\sum_i w_i (y_i-z_i)^2 + \lambda ...
5
votes
3answers
465 views

Shortest distance between two lines in 3-dimensional space [closed]

Can someone explain to me how to solve this question? Find the shortest distance between the lines $L_1 = \left\{t \begin{bmatrix} 1\\ 1\\ 1\end{bmatrix} : t \in \mathbb{R}\right\}$ and $L_2 = ...
-1
votes
0answers
16 views

Condition number plane fit

I fit a 3D plane to 3D points. I setup the corresponding linear system $Ax=0$ by removing the mean of all the points and stacking them as rows into $A$, and solve for a non-trivial ($x\neq0$) using ...
0
votes
1answer
27 views

Complex ($\mathbb C$) least squares derivation

I know how to derive the least squares in the real domain. If a tall matrix $A$ and a column vector $b$ are real, then the solution of the least squares problem $Ax = b$ can be derived as: ...
0
votes
1answer
12 views

Least squares, show estimate is unbiased

I'm struggling with the following problem from Gilbert Strang's book on linear algebra: First assumption behind least squares $Ax=b-e$ where $e$ is noise with mean zero. Multiply the error vectors ...
1
vote
0answers
13 views

Complex Least Squares With Magnitude Equality Constraints

For $\mathbf{x} \in \mathbb{C}^N$, I'd like to solve the following problem: $$ \mathbf{x}^\ast = \arg \min_{\mathbf{x}} \Vert \mathbf{Ax-b} \Vert_2 \,\,\,\,\,\, \mathrm{s.t.} \,\,\,\,\, \Vert x_i ...
0
votes
0answers
34 views

Which curve (surface) is this?

We're having trouble fitting our data... well, we don't even know which function we should fit onto. Anybody knows if this function is well defined mathematically?
0
votes
0answers
10 views

Least-squares solution to almost-linear equations with a few cross-terms

I have an intermediate number of equations N (say, 15) that I'd like to solve in a least-squares manner for M unknowns (M To clarify, I have a set up with unknowns $x_i$like: $y_1 = a_1 x_1 + b_1 ...
0
votes
1answer
13 views

How to shift points optimally for best rounding

I have sets of points. E.g.: 5.664, 2.292, 1.368, 0.18, 3.3, 4.74, 7.812, 6.564, 5.352, 4.008, 2.568, 5.352 I'd like to shift them a bit (add some uniform dx to all of them) to make them closer to ...
1
vote
1answer
36 views

Analytical solution to nonlinear least-squares problem

I have a data set which can be fit well to a single gaussian model, with dependent variables $y_i$ and independent variables $x_i$, with $i=1...N$. I want to avoid using a nonlinear fitting library, ...
0
votes
0answers
6 views

Deriving spectral norm or similar quantity for structured random matrix

I have a problem where I have no idea to start. Suppose a simple Least Squares system with $M$ unknowns $c$ and $N$ observations $y$ which is given through the linear mapping $X$: $$y = X c$$ It is ...
2
votes
1answer
24 views

Properties of the solution of a linear system with random equations

$x_i$ is drawn from $\mathrm{unif}(a,b)$, $y_i$ is drawn from $\mathrm{unif}(c,d)$. $x_i$ are independent from each other. $y_i$ are independent from each other. $x_i$ are independent of $y_i$. $i$ ...
7
votes
0answers
84 views

Does anyone know a reference to best-fitting lines with integral coefficients?

I'm writing up a manual on how to generate "nice" Linear Algebra problems; that is, where the solutions tend to be integral. I "discovered" the following fact about the best-fitting line: Theorem. ...
2
votes
1answer
20 views

Deriving the identity: $\hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}$

For some reason I am having an extremely hard time finding out how the following expression is derived $$ \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} $$ Is ...
0
votes
0answers
14 views

How to use leave-one-out cross-validation scheme to compute the accuracy of a linear model fit

Using the least squares estimation I calculated the model fit for a dataset where: $$ p = \beta_{0} + \beta_{1} * t $$ How could I use leave one out cross-validation(CV) scheme to compute accuracy ...
0
votes
0answers
21 views

Least-squares when some coefficient is $0$?

I'm trying to find least squares approximation $p(x)=c_1x+c_2x^2$ of $f(x)=xe^{x/2}$ in $[0,2]$. Using the algorithm here, p.7.: http://www.math.niu.edu/~dattab/MATH435.2013/APPROXIMATION.pdf I'm ...
1
vote
1answer
19 views

Normal system of the least square method

I'm trying to show the following. $Pa$ is the approximation system of $y$. I want to show that finding the minimmum for the function $$f(a,y)=||Pa-y||_2^2$$ is equivalent to solve the normal system of ...
0
votes
0answers
12 views

Inverse kinematics - How do i compute the du?

I am at the moment trying to implement at jacobian based inverse kinematics solver, which is given a current homogeneous Transformation matrix r(q) and a desired homogenous tranformation matrix ...
0
votes
0answers
17 views

Does this method for solving an overdetermined systemof equations minimize an error?

Suppose we have an overdetermined set of linear equations $X\beta=y$, which we wish to solve for $\beta$. $X$ is an $n\times m$ matrix with $n>m$. The standard method of doing this is, of course, ...
0
votes
0answers
26 views

Image processing, optimization via regularization - efficient strategy

i would like to solve the following system: $J(x) = |Ax-b|_2^2+\gamma|\nabla x|_2^2$ subject to: $x \geq 0, \sum_i x_i = 1$ The underlying problem is to derive the PSF from a sharp and blurry ...
0
votes
0answers
23 views

Number of measurements for least squares and relation to maximum likelihood

I have a simple overdetermined system of equations: $$ y = Xc + e $$ $y, e \in \mathbb{R}^n$, $c \in \mathbb{R}^m$, $X \in \mathbb{R}^{n \times m}$, $e \sim \mathcal{N}(o,\sigma^2)$, $n>>m$ ...
1
vote
1answer
21 views

Understanding Linear Least Squares Problem Statement

My computer vision class is covering linear least squares problems with SVD. In our notes, the problem statement is as follows: min $\mid Ax - b\mid^2 = min(Ax - b)^T(Ax - b)$ = min $x^TA^TAx - ...
1
vote
2answers
14 views

How to simplify OLS formulas?

Let $\lbrace x_i,y_i\rbrace_{i=1}^n$ be a random sample. I am trying to simplify the following expression $$\frac{\sum_{i=1}^n x_i y_i - n \bar{X}\bar{Y}}{\sum_{i=1}^n x_i^2 -n\bar{X}^2}$$ to show ...
1
vote
1answer
26 views

Solving Norm-Constrained Homogeneous Linear Least Squares

I am learning how to solve a norm-constrained homogeneous linear least squares problem. min $(norm(Ax))^2$ for x such that norm(x) = 1 The problem is set up with a Lagrangian as follows: cost = ...
0
votes
0answers
31 views

Residual norm of active set method in non negative least squares

I am having trouble with understanding one of the statements in active set methods done for non negative least squares given in Book written by Lawson. The quadratic problem can be written as ...
0
votes
1answer
46 views

How to plot correct best fit line?

I'm working on a software where I'm plotting graphs and finding best fit line. I have used Least-Square Method and linear regression technique with y = mx + c My problem is that when most of the X ...
0
votes
0answers
32 views

Least Squares with Singular $AA^T$

Given the following system, find all least squares solutions: $\begin{bmatrix}1 & 2 & 3\\2 & 3 & 4\\3 & 4 & 5\end{bmatrix} \vec{x} = \begin{bmatrix}1\\1\\2\end{bmatrix}$ ...
0
votes
0answers
15 views

Solving non-linear functional equations numerically by sequence of linear least-squares?

So I am experimenting with a linear systems solver to find new exciting applications for it. While it is possible to play around to solve some of the more basic functional equations, I am trying to be ...
1
vote
0answers
18 views

Regularization least squares

Given image $x$ and a transformation (blurring) $K$, we get a blurry image $f$. The blurring transformation $K$ is ill conditioned. For a given $f$, find $x$. Solution via regularization: ...
0
votes
1answer
22 views

Fitting a curve of the form Y = ae^(-b) using Least Squares Estimators

I'm trying to fit a set of data into the curve Y=ae^(-bx) There are 6 pairs of data (time,quantity). I have used log on both sides, which gave me ...
0
votes
0answers
10 views

Different ways of solving $\underset{\mathbf{s}}{\text{min}}\;\|F\mathbf{s}-\mathbf{x}\|_{l_2}^2 + \|W\mathbf{s}\|_{l_2}^2$ least square problem?

The problem that I am going to describe arises from compressed sensing technique and after using weighted least squares it can be transformed into the following least squares problem: ...
0
votes
1answer
27 views

Assumptions required for expected value of sum of products to equal zero.

First define $x_i$ is determenistic variable, $v_i$ is random variable. Consider the expression: $$\text E(\sum_{i=1}^{\ n} x_iv_i)= \sum_{i=1}^{\ n} \text E (x_iv_i)=\sum_{i=1}^{\ n} \text E (x_i) ...
1
vote
1answer
31 views

How can I force least square solution matrix to be diagonal?

Let's say I have the following equation $$AX=B$$ where $A$ is a $8\times 3$ matrix (known), $X$ is a $3\times3$ "diagonal" matrix which represents the coefficients (unknown) and $B$ is a ...
0
votes
0answers
20 views

general framework for proof of convexity in least square problem

Let $f(x_i,\theta) : X \times \Theta \rightarrow Z$ be a parametric function with parameter $\theta$ that we wish to fit to set of samples $S =\{(x_i,z_i)\}_{i=1}^N$, where $(x_i, z_i) \in X \times ...
1
vote
1answer
65 views

why do we say SVD can handle singular matrx when doing least square? Comparison of SVD and QR decomposition

I don't quite understand why we say that QR decomposition doesn't handle singular matrix, while SVD does when they are used for least square problem? My example in Matlab seems to support the ...
1
vote
0answers
33 views

How to constrain the linear least squares fit of a quadratic polynomial with known constraints

How to constrain this fit I have some function , $f(x) = a x^2+b x+c$ , with the constraints $a<0$ and $c = \frac{b^2}{4a}+\frac{1}{2}ln(\frac{-a}{\pi})$ I have measured $f(x)$ for some $x$. Can ...
1
vote
0answers
11 views

regularized least squares (L1 norm)

My objective function that is to be minimized is as follows: $$\|y-Ax\|_2^2 + \alpha\|Lx\|_1$$ where $L$ is the gradient operator. Now this problem seems convex because the first term is quadratic ...
3
votes
1answer
45 views

Power curve fitting

Least Squares can be used to fit the following power curve to given data points. $y=ax^b$ where $a,b$ are constants to be determined by the fitting process as seen here. Is there a way to fit ...
1
vote
1answer
29 views

The validity of Least Squares Fitting to a specific problem

I'm going to try and keep this question broad, so I apologise if it's poorly written. I have a series of functions; $$ \Psi_{j} = \sum_{n = 1}^{N} A_{n} \sinh{2 \pi n S_{j}} \cos{2 \pi n X_{j}} + ...