Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere. See https://en.wikipedia.org/wiki/Least_squares

learn more… | top users | synonyms

0
votes
1answer
21 views

Least squares solutions and orthogonal projection?

I found the least squares solution for the following inconsistent system of equations: $ x_1 - x_2 = 0$ $ x_1 + x_2 = 5 $ $-x_1 + x_2 = 2$ , which turned out to be $ \begin{bmatrix} 2\\ 3\\ \end{...
3
votes
3answers
33 views

Smallest possible value of the norm?

The vectors $ \vec{u_1} = \begin{bmatrix} 1 \\ 1 \\ 1\\ 1 \end{bmatrix} $ and $ \vec{u_2} = \begin{bmatrix} 1 \\ -1 \\ 1\\ -1 \end{bmatrix} $ are orthonormal in $ \mathbb{R}^4$....
1
vote
0answers
16 views

Required polynomial order for 2D least square function fit

I am working with a point cloud of approximately 500 points which has the form $p = f(x,y)$ and I need to find a function $\hat{f}$ that will correctly approximate $f$ on all of its domain. To do so, ...
0
votes
0answers
19 views

The Proof of Minimizing Least Square

We Know that To minimize the sum of error (objective Function) $\ J = (y(t)-\theta (t) u(t))^2 $ (eq. 1) is done by using least square : $\theta (t) = \theta (t-1) + \gamma y(\theta ...
1
vote
1answer
17 views

Proving OLS estimator of variance

According to Gujarati, author notes that in a simple linear equation form $Y_i=\alpha +\beta X_i + \epsilon_i$ where regression model is defined as $\hat Y_i =\hat \alpha + \hat \beta X_i$ OLS method ...
0
votes
1answer
21 views

The OLS Estimator of $\sigma^2$

I have a question regarding the OLS Estimator of $\sigma^2$. In Gujarati's book on Econometrics author derives $E(\sum_{i=1}^n \hat u_i^2)$ (aka the expected value of residuals) to be $(n-2)\sigma$. ...
1
vote
1answer
37 views

Solving a homogenous, weighted LSE problem

I search the non trivial solution for the system $\bf{Ax}=\bf{b} = \bf{0}$, where the equations are weighted with the matrix $\bf{W}$. I found 2 different approaches for each part, which I would ...
0
votes
0answers
11 views

how to minimize $tr (X^T W - Y^T W_p)(X^T W -Y^TW_p)^T$ in closed form

Assume we are dealing with matrices. Then how to minimize $$ E(W,W_p) = tr (X^T W - Y^T W_p)(X^T W -Y^TW_p)^T $$ w.r.t both $W, W_p$ simultaneously? I can calculate the derivatives of $W$ and $W_p$ ...
2
votes
2answers
52 views

Proof related to the least squares method

I've seen this exercise in several statistics text, but how they get to the final formula is something that I don't quite get. How do two squared terms suddenly become a binomial term? I've been ...
0
votes
2answers
60 views

Solve an overdetermined system of linear equations

I have doubt to solve this system of equations \begin{cases} x+y=r_1\\ x+z=c_1\\ x+w=d_1\\ y+z=d_2\\ y+w=c_2\\ z+w=r_2 \end{cases} Is it an overdetermined system because I see there are more ...
1
vote
0answers
23 views

Explicit least-squares method for horizontal shifts of a function

I have a sequence of $N$ strictly positive real values $y_n$. They form some kind of peak; for simplicity, let's assume $f(x, \mu) = A \exp^{-(x-\mu)^2}$ is the shape, with $A$ and $\mu$ real (in the ...
1
vote
2answers
15 views

2-normed square meaning

I heard $2$-normed square in a lecture talking about the objective function of least-squares. What does the $2$ mean? I understand we take norm and square it, $2$ doesn't make sense to me. $$\|Ax−B\|^...
0
votes
0answers
20 views

Detecting highly oscillatory polynomials

Given a set of known values $\phi_1 \ldots \phi_n$ located at points $(x_1, y_1) \ldots (x_n, y_n)$, I want to approximate the value $\phi_0$ at the origin $(0,0)$. To do this, I am using a least ...
1
vote
1answer
22 views

Average Percent Rate of Change

Excuse the png equations, still a MathJax newbie. I am analyzing data I have computed: Alcohol content and Caffeine content retention after a duration of 8 hours for each. I had gotten the data in ...
-1
votes
1answer
38 views

Finding the global minimum

Let $f~:~\Bbb R^2\to \Bbb R$ be defined as: $$f(x)=\left\|\begin{bmatrix}2&1\\3&1\\4&2\end{bmatrix}\begin{bmatrix}x_1\\x_2\end{bmatrix} - \begin{bmatrix}2\\1\\7\end{bmatrix}\right\|_2^2$$ ...
1
vote
1answer
32 views

Prove the invertibility of $X^T X$ when $X$ is a (rectangular) Toeplitz-like matrix.

In order to use a minimum squares estimator over some discrete dynamic system parameters, it is necessary to prove that the product $X^T X$ is invertible. Consider the following $N$ by $n+1$ matrix $X$...
0
votes
0answers
24 views

Find rotation matrix to match points in parallel projection

I am given two sets of 3D points (actually 2D, see below) with corresponding pairs. I am seeking two 3D rotation matrices, such that (only) the X and Y components of the rotated points match best (...
-1
votes
1answer
10 views

Least squares method what is an good error [closed]

Hi I have started using the least squares method and I want to calculate the mean error of my approximation. I use the following formular for calculating the error: $\ \mathcal E = ||A\hat x - y||/\...
2
votes
1answer
31 views

Is the least-squares solution unique?

I am looking for a line closest to $(-5, -2)$, $(-2, 0)$, $(-1, 0)$, $(2, 3)$, $(5, 4)$ using the least square solution. So I set the line as $$ax+by+c=0$$ let $a=1$ (where $a$ is not $0$ obviously) ...
1
vote
1answer
26 views

Derivative of dot product of Residual Sum Square in matrix notation

I am trying to derive the following expression w.r.t. $\beta$: \begin{equation} RSS(\beta) = (\mathbf{y} - \mathbf{X} \beta)^T (\mathbf{y} - \mathbf{X} \beta) \end{equation} I know that the ...
2
votes
1answer
31 views

Least Squares Sensitivity to data

Let ($x_1$,$y_1$),...,($x_n$,$y_n$) be my data set. I have a function $f(x,{\bf c})$ where ${\bf c}=(c_1,...,c_m)$ is a vector of $m$ parameters. I want to fit to the data using non-linear least ...
1
vote
0answers
17 views

Confused about solution to the piecewise constant regression model

I am confused about the solution to the following solution to fitting piecewise constants: Specifically, are we minimising the sum of squares, that is, finding the vector $\beta = (\beta_1,\beta_2, ...
0
votes
1answer
20 views

error term in time-series Seasonal AR model

I am reading a paper related to timeseries forecasting in which I have a question regarding the seasonal AR model described in equation (1.2) namely: $log(y_t)$~$log(y_{t-1}) + log(y_{t-12}) + x^{(1)}...
1
vote
3answers
59 views

Distance from a set to a point

There is this exercise I cannot understand well. It asks me for the distance between this set in $\mathbb{R}^3$ $$U = \{(x, y, z)\ |\ ax + y - 2z = 0, z = 0 \}$$ and the point $(0, b, 1)$. Also it ...
1
vote
1answer
37 views

How was this least squares polynomial obtained (solution provided)

Obtain a least squares polynomial of degree 2. $$ \begin{array}{c|lcr} x & \text{0} & \text{0.25} & \text{0.5}& \text{0.75}&\text{1} \\ \hline y & 2.9646 & 3.1826 & 3....
1
vote
2answers
46 views

interpreting the effect of transpose in the normal equations

I have a question about the normal equation. $A$ an $m\times n$ matrix with trivial nullspace, $y$ a vector outside the range of $A$. The vector $x$ that minimizes $|| Ax - y ||^2$ is the solution to $...
2
votes
1answer
199 views

Least-squares problem with quadratic equality constraint

I want to find the solution of a Lagrange equation whose inputs are matrices. First I have the equation Ax=0. By decomposing $A$ into $A_3$ (columns 9 to 11 of A), $A_9$ (the rest of the columns), ...
0
votes
0answers
16 views

Weighted Linear Regression

I am performing linear regression analysis on a time-series of data. Data contains some missing values, My question is, if I impute the missing values using mean of all the values and I want to ...
3
votes
0answers
44 views

SVD as a solution to linear least squares

I'm a little confused about the various explanations for using Singular Value Decomposition (SVD) to solve the Linear Least Squares (LLS) problem. I understand that LLS attempts fit $Ax=b$ by ...
1
vote
2answers
55 views

Given a set of vectors and a target vector, find the set of scaling factors that minimizes distance of sum of those vectors from target

I have a set of $n$ starting vectors $\vec i_n$ and a target vector $\vec t$. I have a set of scaling factors $a_n$ for which I can compute the sum $\vec s$: $$ \vec s = \sum_{i=1}^n {a_i \vec i_i} $$...
0
votes
1answer
99 views

Solving 3x3 Matrix Q using Nonlinear Least Squares or Cholesky Decomposition

I am trying to solve a system of equations using Cholesky decomposition. I would like to solve for the 3x3 matrix Q given: $\hat{i_f}^t Q Q^t \hat{i_f} = 1 $ $ \hat{j_f}^t Q Q^t \hat{j_f} = 1 $ $...
0
votes
0answers
22 views

How to calculate a projection matrix for nonnegative constrained least squares?

Suppose we have a data vector $\boldsymbol{z}$ in R^{p} and a training data matrix $\boldsymbol{X}$ in $R^{p \times N}$, where N (N>p) is the number of samples in the training data matrix. If we'd ...
2
votes
1answer
24 views

Variance of Least Squares Estimator

Suppose a fit a line using the method of least squares to $n$ points, all the standard statistical assumptions hold, and I want to estimate that line at a new point, $x_0$. Denoting that value by $\...
3
votes
2answers
69 views

Least Squares Alternates- approximating functions

I was given this least squares problem to solve: Find a linear function $\ell(x)$ such that $\displaystyle\int_0^1(e^x-\ell(x))^2{\rm d}x$ is minimized. As an answer, I got $\ell(x)=0.5876+0....
1
vote
1answer
37 views

Linerar Regression ( Least square fit)

The problem is given below: Simultaneous values of time $t$ and output $y$ from a specific sensor has been measured and is tabulated below $$\begin{array}{cc} t & y \\ \hline 1 & 17 \\ ...
0
votes
0answers
34 views

recursive least squares for nearly singular matrices

I have an image reconstruction problem which I want to solve as a linear system $Ax=y$. A matrix is big, but for the beginning I can shrink the imaging region to $nPix$ = 2000 pixels. number of ...
1
vote
1answer
69 views

Weight Modification for Computationally-Efficient Nonlinear Least Squares Optimization

There was a time where I could figure this out for myself, but my math skills are rustier than I thought, so I have to humbly beg for help. Thank you in advance. I am solving a weighted nonlinear ...
0
votes
1answer
26 views

Least squares without solution

Talking about simple linear regression (k=1), in which cases the Normal Equations have unique solution? And infinite? And when the Normal equations have no solution?
0
votes
0answers
26 views

mean-deviation form, why orthogonal?

This is from my textbook Why the column of the new design matrix are orthogonal? for example, let say $A=\begin{pmatrix} 1& 1& 4\\ 1& 2& 0\\ 1& 3& 2 \end{pmatrix}$ ...
0
votes
0answers
20 views

QR factorization for least squares

This is from my textbook I don't undertand why small errorr in $A^TA$ can lead to large error in cofficient matrix? Because A=QR, so there should be no difference to use A or QR anyway.Could someone ...
0
votes
1answer
19 views

How to expand the matrix in Matlab (forming $A$ matrix in least square problem)

If I have 5 data points (actually, this could be vary huge number): $a = [1;2;3;4;5]$ To each data point, I can form a matrix $[a(i),2a(i)]$. (actually, I have to find the matrix, this matrix needs ...
0
votes
0answers
9 views

Aligning matrices, normalization. Calculating coefficients.

So as a pre-task for my upcoming exam this is one of the rehearsal assignments. I can't wrap my head around this one at all, haven't seen anything like it earlier, and I can't seem to find any ...
0
votes
1answer
31 views

Least Squares Revisited

I am reading a paper on regression and there seems to be a simple substitution but I just cannot get my head around it. My question is how you go from equation (3) and (4) to (5)? Please let me know ...
0
votes
0answers
21 views

Efficient estimator: best choice for the weight matrix in the Weighted Least Squares Estimation

I am facing the linear regression problem in the form: $$y = \Phi\theta+\eta$$ where $y\in\mathbb{R}^N$ is the vector of the measurements (the available data), $\Phi\in\mathbb{R}^{N\times n}$ the ...
1
vote
1answer
62 views

Finding the normal equation

After my semester at Umich my mathematics professor issued me an abundance of problems to keep my head in the game during the summer. One of the questions which threw me off was finding the normal ...
0
votes
1answer
34 views

How to fit a set of 3D points to a helical curve?

suppose I have a set of points in $\mathbb{R}^3$, and I want to find an arbitrary helix which best approximates these points. An arbitrary helix in $\mathbb{R}^3$ can be parametrized as $$\vec{r}(t)...
0
votes
0answers
17 views

Finding the approximate confidence interval for a paramter

Intro From the previous exercise, we were instructed to find a parameter $b$ using maximum likelihood and least square methods in the Rayleigh distribution, given by $$ {f_X}(x) = \frac{x}{{{...
1
vote
1answer
26 views

Asymmetric Least Squares Conversion from Equation to Matrix

In solving for asymmetric least squares baseline correction as defined in the article by Eilers and Boelens, the general equation is defined as: $$S = \displaystyle\sum_i w_i (y_i-z_i)^2 + \lambda \...
6
votes
3answers
486 views

Shortest distance between two lines in 3-dimensional space [closed]

Can someone explain to me how to solve this question? Find the shortest distance between the lines $L_1 = \left\{t \begin{bmatrix} 1\\ 1\\ 1\end{bmatrix} : t \in \mathbb{R}\right\}$ and $L_2 = \left\...
0
votes
1answer
28 views

Complex ($\mathbb C$) least squares derivation

I know how to derive the least squares in the real domain. If a tall matrix $A$ and a column vector $b$ are real, then the solution of the least squares problem $Ax = b$ can be derived as: $$\begin{...