Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere. See https://en.wikipedia.org/wiki/Least_squares

learn more… | top users | synonyms

0
votes
0answers
6 views

Prove the invertibility of $X^T X$ when $X$ is a (rectangular) Toeplitz-like matrix.

In order to use a minimum squares estimator over some discrete dynamic system parameters, it is necessary to prove that the product $X^T X$ is invertible. Consider the following $N$ by $n+1$ matrix $X$...
0
votes
0answers
21 views

Find rotation matrix to match points in parallel projection

I am given two sets of 3D points (actually 2D, see below) with corresponding pairs. I am seeking two 3D rotation matrices, such that (only) the X and Y components of the rotated points match best (...
-1
votes
1answer
10 views

Least squares method what is an good error [on hold]

Hi I have started using the least squares method and I want to calculate the mean error of my approximation. I use the following formular for calculating the error: $\ \mathcal E = ||A\hat x - y||/\...
2
votes
1answer
29 views

Is the least-squares solution unique?

I am looking for a line closest to $(-5, -2)$, $(-2, 0)$, $(-1, 0)$, $(2, 3)$, $(5, 4)$ using the least square solution. So I set the line as $$ax+by+c=0$$ let $a=1$ (where $a$ is not $0$ obviously) ...
1
vote
1answer
25 views

Derivative of dot product of Residual Sum Square in matrix notation

I am trying to derive the following expression w.r.t. $\beta$: \begin{equation} RSS(\beta) = (\mathbf{y} - \mathbf{X} \beta)^T (\mathbf{y} - \mathbf{X} \beta) \end{equation} I know that the ...
2
votes
1answer
29 views

Least Squares Sensitivity to data

Let ($x_1$,$y_1$),...,($x_n$,$y_n$) be my data set. I have a function $f(x,{\bf c})$ where ${\bf c}=(c_1,...,c_m)$ is a vector of $m$ parameters. I want to fit to the data using non-linear least ...
1
vote
0answers
16 views

Confused about solution to the piecewise constant regression model

I am confused about the solution to the following solution to fitting piecewise constants: Specifically, are we minimising the sum of squares, that is, finding the vector $\beta = (\beta_1,\beta_2, ...
0
votes
1answer
19 views

error term in time-series Seasonal AR model

I am reading a paper related to timeseries forecasting in which I have a question regarding the seasonal AR model described in equation (1.2) namely: $log(y_t)$~$log(y_{t-1}) + log(y_{t-12}) + x^{(1)}...
1
vote
3answers
58 views

Distance from a set to a point

There is this exercise I cannot understand well. It asks me for the distance between this set in $\mathbb{R}^3$ $$U = \{(x, y, z)\ |\ ax + y - 2z = 0, z = 0 \}$$ and the point $(0, b, 1)$. Also it ...
1
vote
1answer
37 views

How was this least squares polynomial obtained (solution provided)

Obtain a least squares polynomial of degree 2. $$ \begin{array}{c|lcr} x & \text{0} & \text{0.25} & \text{0.5}& \text{0.75}&\text{1} \\ \hline y & 2.9646 & 3.1826 & 3....
1
vote
2answers
46 views

interpreting the effect of transpose in the normal equations

I have a question about the normal equation. $A$ an $m\times n$ matrix with trivial nullspace, $y$ a vector outside the range of $A$. The vector $x$ that minimizes $|| Ax - y ||^2$ is the solution to $...
2
votes
0answers
163 views

Least-squares problem with quadratic equality constraint

I want to find the solution of a Lagrange equation whose inputs are matrices. First I have the equation Ax=0. By decomposing $A$ into $A_3$ (columns 9 to 11 of A), $A_9$ (the rest of the columns), ...
0
votes
0answers
13 views

Weighted Linear Regression

I am performing linear regression analysis on a time-series of data. Data contains some missing values, My question is, if I impute the missing values using mean of all the values and I want to ...
3
votes
0answers
39 views

SVD as a solution to linear least squares

I'm a little confused about the various explanations for using Singular Value Decomposition (SVD) to solve the Linear Least Squares (LLS) problem. I understand that LLS attempts fit $Ax=b$ by ...
1
vote
2answers
52 views

Given a set of vectors and a target vector, find the set of scaling factors that minimizes distance of sum of those vectors from target

I have a set of $n$ starting vectors $\vec i_n$ and a target vector $\vec t$. I have a set of scaling factors $a_n$ for which I can compute the sum $\vec s$: $$ \vec s = \sum_{i=1}^n {a_i \vec i_i} $$...
0
votes
1answer
94 views

Solving 3x3 Matrix Q using Nonlinear Least Squares or Cholesky Decomposition

I am trying to solve a system of equations using Cholesky decomposition. I would like to solve for the 3x3 matrix Q given: $\hat{i_f}^t Q Q^t \hat{i_f} = 1 $ $ \hat{j_f}^t Q Q^t \hat{j_f} = 1 $ $...
0
votes
0answers
20 views

How to calculate a projection matrix for nonnegative constrained least squares?

Suppose we have a data vector $\boldsymbol{z}$ in R^{p} and a training data matrix $\boldsymbol{X}$ in $R^{p \times N}$, where N (N>p) is the number of samples in the training data matrix. If we'd ...
2
votes
1answer
24 views

Variance of Least Squares Estimator

Suppose a fit a line using the method of least squares to $n$ points, all the standard statistical assumptions hold, and I want to estimate that line at a new point, $x_0$. Denoting that value by $\...
3
votes
2answers
67 views

Least Squares Alternates- approximating functions

I was given this least squares problem to solve: Find a linear function $\ell(x)$ such that $\displaystyle\int_0^1(e^x-\ell(x))^2{\rm d}x$ is minimized. As an answer, I got $\ell(x)=0.5876+0....
1
vote
1answer
37 views

Linerar Regression ( Least square fit)

The problem is given below: Simultaneous values of time $t$ and output $y$ from a specific sensor has been measured and is tabulated below $$\begin{array}{cc} t & y \\ \hline 1 & 17 \\ ...
0
votes
0answers
29 views

recursive least squares for nearly singular matrices

I have an image reconstruction problem which I want to solve as a linear system $Ax=y$. A matrix is big, but for the beginning I can shrink the imaging region to $nPix$ = 2000 pixels. number of ...
1
vote
1answer
69 views

Weight Modification for Computationally-Efficient Nonlinear Least Squares Optimization

There was a time where I could figure this out for myself, but my math skills are rustier than I thought, so I have to humbly beg for help. Thank you in advance. I am solving a weighted nonlinear ...
0
votes
1answer
26 views

Least squares without solution

Talking about simple linear regression (k=1), in which cases the Normal Equations have unique solution? And infinite? And when the Normal equations have no solution?
0
votes
0answers
25 views

mean-deviation form, why orthogonal?

This is from my textbook Why the column of the new design matrix are orthogonal? for example, let say $A=\begin{pmatrix} 1& 1& 4\\ 1& 2& 0\\ 1& 3& 2 \end{pmatrix}$ ...
0
votes
0answers
20 views

QR factorization for least squares

This is from my textbook I don't undertand why small errorr in $A^TA$ can lead to large error in cofficient matrix? Because A=QR, so there should be no difference to use A or QR anyway.Could someone ...
0
votes
1answer
18 views

How to expand the matrix in Matlab (forming $A$ matrix in least square problem)

If I have 5 data points (actually, this could be vary huge number): $a = [1;2;3;4;5]$ To each data point, I can form a matrix $[a(i),2a(i)]$. (actually, I have to find the matrix, this matrix needs ...
0
votes
0answers
9 views

Aligning matrices, normalization. Calculating coefficients.

So as a pre-task for my upcoming exam this is one of the rehearsal assignments. I can't wrap my head around this one at all, haven't seen anything like it earlier, and I can't seem to find any ...
0
votes
1answer
30 views

Least Squares Revisited

I am reading a paper on regression and there seems to be a simple substitution but I just cannot get my head around it. My question is how you go from equation (3) and (4) to (5)? Please let me know ...
0
votes
0answers
19 views

Efficient estimator: best choice for the weight matrix in the Weighted Least Squares Estimation

I am facing the linear regression problem in the form: $$y = \Phi\theta+\eta$$ where $y\in\mathbb{R}^N$ is the vector of the measurements (the available data), $\Phi\in\mathbb{R}^{N\times n}$ the ...
1
vote
1answer
62 views

Finding the normal equation

After my semester at Umich my mathematics professor issued me an abundance of problems to keep my head in the game during the summer. One of the questions which threw me off was finding the normal ...
0
votes
1answer
33 views

How to fit a set of 3D points to a helical curve?

suppose I have a set of points in $\mathbb{R}^3$, and I want to find an arbitrary helix which best approximates these points. An arbitrary helix in $\mathbb{R}^3$ can be parametrized as $$\vec{r}(t)...
0
votes
0answers
17 views

Finding the approximate confidence interval for a paramter

Intro From the previous exercise, we were instructed to find a parameter $b$ using maximum likelihood and least square methods in the Rayleigh distribution, given by $$ {f_X}(x) = \frac{x}{{{...
1
vote
1answer
24 views

Asymmetric Least Squares Conversion from Equation to Matrix

In solving for asymmetric least squares baseline correction as defined in the article by Eilers and Boelens, the general equation is defined as: $$S = \displaystyle\sum_i w_i (y_i-z_i)^2 + \lambda \...
5
votes
3answers
476 views

Shortest distance between two lines in 3-dimensional space [closed]

Can someone explain to me how to solve this question? Find the shortest distance between the lines $L_1 = \left\{t \begin{bmatrix} 1\\ 1\\ 1\end{bmatrix} : t \in \mathbb{R}\right\}$ and $L_2 = \left\...
0
votes
1answer
28 views

Complex ($\mathbb C$) least squares derivation

I know how to derive the least squares in the real domain. If a tall matrix $A$ and a column vector $b$ are real, then the solution of the least squares problem $Ax = b$ can be derived as: $$\begin{...
0
votes
1answer
13 views

Least squares, show estimate is unbiased

I'm struggling with the following problem from Gilbert Strang's book on linear algebra: First assumption behind least squares $Ax=b-e$ where $e$ is noise with mean zero. Multiply the error vectors $e=...
1
vote
0answers
17 views

Complex Least Squares With Magnitude Equality Constraints

For $\mathbf{x} \in \mathbb{C}^N$, I'd like to solve the following problem: $$ \mathbf{x}^\ast = \arg \min_{\mathbf{x}} \Vert \mathbf{Ax-b} \Vert_2 \,\,\,\,\,\, \mathrm{s.t.} \,\,\,\,\, \Vert x_i \...
0
votes
0answers
35 views

Which curve (surface) is this?

We're having trouble fitting our data... well, we don't even know which function we should fit onto. Anybody knows if this function is well defined mathematically?
0
votes
0answers
11 views

Least-squares solution to almost-linear equations with a few cross-terms

I have an intermediate number of equations N (say, 15) that I'd like to solve in a least-squares manner for M unknowns (M To clarify, I have a set up with unknowns $x_i$like: $y_1 = a_1 x_1 + b_1 x_3$...
0
votes
1answer
14 views

How to shift points optimally for best rounding

I have sets of points. E.g.: 5.664, 2.292, 1.368, 0.18, 3.3, 4.74, 7.812, 6.564, 5.352, 4.008, 2.568, 5.352 I'd like to shift them a bit (add some uniform dx to all of them) to make them closer to ...
1
vote
1answer
41 views

Analytical solution to nonlinear least-squares problem

I have a data set which can be fit well to a single gaussian model, with dependent variables $y_i$ and independent variables $x_i$, with $i=1...N$. I want to avoid using a nonlinear fitting library, ...
0
votes
0answers
8 views

Deriving spectral norm or similar quantity for structured random matrix

I have a problem where I have no idea to start. Suppose a simple Least Squares system with $M$ unknowns $c$ and $N$ observations $y$ which is given through the linear mapping $X$: $$y = X c$$ It is ...
2
votes
1answer
26 views

Properties of the solution of a linear system with random equations

$x_i$ is drawn from $\mathrm{unif}(a,b)$, $y_i$ is drawn from $\mathrm{unif}(c,d)$. $x_i$ are independent from each other. $y_i$ are independent from each other. $x_i$ are independent of $y_i$. $i$ ...
7
votes
0answers
84 views

Does anyone know a reference to best-fitting lines with integral coefficients?

I'm writing up a manual on how to generate "nice" Linear Algebra problems; that is, where the solutions tend to be integral. I "discovered" the following fact about the best-fitting line: Theorem. ...
2
votes
1answer
22 views

Deriving the identity: $\hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}$

For some reason I am having an extremely hard time finding out how the following expression is derived $$ \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} $$ Is ...
0
votes
0answers
16 views

How to use leave-one-out cross-validation scheme to compute the accuracy of a linear model fit

Using the least squares estimation I calculated the model fit for a dataset where: $$ p = \beta_{0} + \beta_{1} * t $$ How could I use leave one out cross-validation(CV) scheme to compute accuracy ...
0
votes
0answers
21 views

Least-squares when some coefficient is $0$?

I'm trying to find least squares approximation $p(x)=c_1x+c_2x^2$ of $f(x)=xe^{x/2}$ in $[0,2]$. Using the algorithm here, p.7.: http://www.math.niu.edu/~dattab/MATH435.2013/APPROXIMATION.pdf I'm ...
1
vote
1answer
20 views

Normal system of the least square method

I'm trying to show the following. $Pa$ is the approximation system of $y$. I want to show that finding the minimmum for the function $$f(a,y)=||Pa-y||_2^2$$ is equivalent to solve the normal system of ...
0
votes
0answers
12 views

Inverse kinematics - How do i compute the du?

I am at the moment trying to implement at jacobian based inverse kinematics solver, which is given a current homogeneous Transformation matrix r(q) and a desired homogenous tranformation matrix r(q+dq)...
0
votes
0answers
19 views

Does this method for solving an overdetermined systemof equations minimize an error?

Suppose we have an overdetermined set of linear equations $X\beta=y$, which we wish to solve for $\beta$. $X$ is an $n\times m$ matrix with $n>m$. The standard method of doing this is, of course, ...