Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

How do you find the asymptotic distribution of the OLS estimator in a simple linear regression model with an intercept?

How do I find the asymptotic distribution of $\beta_1$ in the model $y_i=\beta_0+\beta_1x_i+\epsilon_i$? I am able to follow Cameron's derivation of a very similar model without an intercept, but when ...
0
votes
0answers
12 views

Legacy code solving least squares adjustments

I was given the task to maintain an old library that, among other things, claims to calculate least squares adjustments. I have read and understood some theory behind it, however I still have troubles ...
1
vote
1answer
24 views

Finding the right optimisation algorithm for nonlinear ARX Model

I am trying to explain my problem in a simplified way: (P1) In the past I have solved the following optimisation problem using the Levenberg–Marquardt algorithm (LVM) offline: Output: $v_e$ Input: i ...
1
vote
1answer
20 views

Losing symmetry in least squares approximation

I tried to find the best linear fit to $(1,-1)$, $(-1,1)$, $(-2,-2)$, and $(2,2)$. There is symmetry within the four points: across $y=x$ and $y=-x$. However, the least squares linear approximation ...
0
votes
1answer
17 views

Kronecker product identity when multiplied by two vectors?

I am interested in least-squares optimization for problems with space-time separable prior state covariances and am trying to break down the quadratic cost function into respective space-time ...
1
vote
0answers
31 views

Optimality conditions of LASSO

In this paper, on page 1122, it states that the optimality conditions for the LASSO give $\hat{\beta} = n_{\lambda}(\hat{\beta} - X^T(X\hat{\beta} - y))$, where $n_\lambda$ is the soft-thresholding ...
1
vote
1answer
63 views

Algorithm to find minimum of sum of squares

I need find the set of integers $X = \{x_0, x_1, ...,x_n\}$ that minimizes: $$\sum_{i=0}^n \left(x_i - T\frac{W_i}{P_i}\right)^2$$ where: $\sum_{i=0}^n x_iP_i < T$ $x_0, x_1, ...,x_n$ are integers ...
0
votes
0answers
16 views

How can I derive OLS predicted error term $\hat{e}_i$ as a function of $e_i$?

First of all, I'd like to say that any kind of help would be really helpful, whether it's a hint or a good grad/undergrad book. Right now I'm working with Econometric Analysis of Cross Section and ...
1
vote
0answers
45 views

Upper bound for the error of the gaussian quadrature$\int_{0}^{1} \log (1+\operatorname{sin} x) \mathrm{d} x$

I'm given the integral $\mathrm{I}=\int_{0}^{1} \log (1+\operatorname{sin} x) \mathrm{d} x$. Through the formula of Gaussian quadrature for 3 points, I can find an approximation to this integral. The ...
0
votes
0answers
16 views

Question regarding Least Squares Algorithm residuals

I was reading about Least Squares and there was an example, however there was a part of the code which I did not understand why we do that step mathematically-wise. (The code was in Matlab). First of ...
1
vote
0answers
25 views

Variable transformation for a multiple linear regression model

I can only transform C) in a way that leaves me with only constants for the parameters. In all other functions I either end up with $ln(\beta_1)$ or need to take the square root of $\beta$, etc. I'm ...
3
votes
2answers
62 views

Gauss-Newton normal equations with norm of residual

The Wiki definition of Gauss-Newton has the following scalar cost function: ${\displaystyle S({\boldsymbol {\beta }})=\sum _{i=1}^{m}r_{i}^{2}({\boldsymbol {\beta }}).}$ where $r_i(\beta)$ are scalar ...
0
votes
1answer
25 views

How to show that an optimization problem is equivalent to the Least Squares problem?

I hope you guys can lend me a hand with this one. Let $A\in \mathbb{R}^{m\times{n}}$ and $b\in \mathbb{R}^{m}$, and consider the following optimization problem: $\min_{x\in\mathbb{R}^{n}} \max_{y\in\...
0
votes
0answers
33 views

How to find minimal solution with the matrix that has no solution

Recently, I learned about the least square solution. And I have leraned a lot from it. The matrix may have infinite least square solutions in the matrix if the matrix isn't full rank. But what if the ...
2
votes
0answers
39 views

Enforce symmetric positive definiteness on least-square matrix equation

Suppose we have two real matrices $A$ and $B$ of size $n\times m$ with $m < n$, and suppose we look for a (small) matrix $X$ of size $m \times m$ such that $\left\|AX -B\right\|_F$ is minimized, ...
0
votes
0answers
20 views

Difference / similarities between Tikhonov's regularization and the least squares method?

What is the difference / similarities between Tikhonov's regularization and the least squares method? I have tried to find information on both but can´t find any clear answers to this question, so I ...
0
votes
0answers
16 views

Nonlinear Regression, least squares

I am trying to solve a non-linear least squares problem like this. $$g(\sum _{1\le j\le J}c_jx_j^i) - f_{mod(i, q)} = y_i\text{ }(1\le i\le I)$$ We want to find $c_j$'s and $f_i$'s where $x^i_j $, $...
0
votes
0answers
25 views

Solving a linear system with rotation, translation and scaling

I have a series of measurements of 3D rotations ($Q_i$) and vectors ($m_i\in\mathbb{R}^3$). I think they fit on the following model: $$Q_if(m_i)\approx n \quad (n \in \mathbb{R}^3)$$ $$f(m_i)=(m_i - c)...
0
votes
0answers
21 views

Least square fitting for sum of exponential

I am finding a way to solve the quadratic constraints non-linear least-square fitting problem. I have a dataset $(x_1^{(i)} ,x_2^{(i)},x_3^{(i)},y^{(i)}) $ and want to find the $\mu_1, \mu_2, \mu_3, \...
0
votes
0answers
13 views

Least Squares Optimization Converging on wrong solution

I'm trying to calculate the position of a multi-constellation GNSS receiver using GPS and GLONASS satellites using least-squares optimization. Sparing the details, I have 5 equations to solve for 5 ...
0
votes
0answers
11 views

analytical derivation of maximum likelihood function of a weighted linear regression with 3D panel data.

I have some trouble coming to analytical solutions in the following MLE problem: \begin{multline} \max _{\alpha, \beta, \gamma, \sigma_{\varepsilon}} \left( \sum_{i=1}^{N} \hat{w}_{i} \sum_{j=1}^...
2
votes
0answers
32 views

Singular Value Decomposition vs iterative Methods for solving inverse Problems

Background: I am trying to motivate why I use an SVD instead of an iterative LMS solver for the solution of an equation of the form $$Ax=b$$ where $A\in\mathbb{C}^{M\times M}$,$x\in\mathbb{C}^{M}$ and ...
0
votes
0answers
40 views

SVD in least-squares problem

I have the following problem $$ W^* := \arg\min_W \| WX - Y \|_F $$ where $W^TW = I$ and $I$ is the identity matrix. Instead of making yet another regression problem we can find optimal orthogonal ...
3
votes
1answer
105 views

Prove that the phase retrieval problem is non-convex

The phase retrieval problem consists of recovering phase information from given intensity measurements, as shown in the image below from Deep phase retrieval: analyzing over-parameterization in phase ...
0
votes
1answer
22 views

Least square method via calculus

Let $f:R^n\to R$ be $f(x)=||Ax-b||$. Prove that $x \in R^n$ where $f$ is minimum meets equations $A^TAx=A^T b$. Any solutions? I have tried to count derivate from it but I could not get any result.
1
vote
0answers
27 views

Uniqueness of Iteratively Reweighted Least Squares

This is a more general question. Suppose I have a heteroscedastic data set that I'd like to fit using IRLS (iteratively re-weighted least squares) to determine $a,b,c,d$. E.g. $f_{model}(x) \approx ...
1
vote
0answers
22 views

Estimate $H_3$ with a Polynomial - Ax^2 + Bx + C

I have the Inner product: $ <f, f> = \sqrt{\int_{-\infty}^{\infty} f(x)^2 \cdot e^{-x^2}}$ and I want to estimate $H_3 = 8x^3 - 12x$ with a function $g(x) = ax^2 + bx + c$ such that the norm $||...
0
votes
1answer
37 views

What's wrong with solving the least-squares problem here? (Simple question)

A naive question that puzzles me a lot: I have $n$ two-dimentional data points $(z_i,w_i)_{i=1}^{n}$ and I want to regress $(z_i)$ on $(w_i)$ by solving the standard least-squares problem: $$ \min_{m,\...
0
votes
0answers
16 views

Optimization problem given weight matrix

Suppose there is a unit norm vector $x \in R^n$, and $y$ is a linear combination of elements of $x$, and let $y = \sum_{i=1}^n w_i^3 x_i$. Given the knowledge of $w_i$, $w = [w_1, ..., w_n]^T$, are we ...
0
votes
2answers
53 views

Closed form solution of $\displaystyle\arg\min_{\alpha \in \Bbb R} \|X - \alpha Y\|_{\text F}^2$

Given $m \times n$ matrices $X$ and $Y$, I am interested in the following least-squares problem. $$\hat \alpha := \arg\min_{\alpha \in \Bbb R} \|X - \alpha Y\|_{\text F}^2$$ Is there any way to ...
0
votes
1answer
69 views

Why doesn't the Gauss-Newton method diverge around the minimum?

I am having trouble visualizing the convergence of the Gauss-Newton method. Consider the simple function of $f(x) = x^2 + 1$. If I try to use the Gauss-Newton method to find the minimum of $f(x)^2$ (...
0
votes
0answers
27 views

Least Squares Solution with Cross Products

Is there a way to find a least squares solution for a vector using a system of cross product equations? For example $\vec{A}, \vec{B}, \vec{C}, \vec{D}$ are all known quantities in 3D space: $$\vec{A} ...
0
votes
1answer
26 views

Find non unique solutions to pseudo inverse least square estimation

For the equation $Ax = B$, I can use the pseudo inverse of $A * B$ to get the best estimate for $x$. Now $A$ is not full rank and there's linearly dependent columns, so when performing $\operatorname{...
0
votes
0answers
36 views

Least Squares solution for a musical note

I'm trying to build a simple least-squares denoiser using linear algebra. Given a note in form A*sin(2πft) and a set of noisy sine data I'm trying to figure out a way to create a Ax = b form so I may ...
0
votes
0answers
28 views

Hessian-free preconditioner for non-linear least squares

I am solving a nonlinear least squares problem using Gauss Newton method. Due to the large dimension of the problem, I use the Hessian-free approach. As a linear solver I use either MINRES or CG. To ...
0
votes
1answer
32 views

How does $\ x_i = (x_i - \overline{x})$ in the proof for ordinary least squares regression?

I am trying to understand how this: $\sum_{i = 1}^{n} x_i(y_i - \overline{y})-\beta_1\sum_{i = 1}^{n} x_i(x_i - \overline{x})$ Could possibly simplify to this: $\sum_{i = 1}^{n} (x_i - \overline{x})(...
1
vote
0answers
27 views

How to find the analytical expression for the measured step response of a dynamic system?

Let's say I have following situation. I have recorded a step response of a dynamic system in following form The data have been gathered with the sampling period $100\,\mu s$. My goal is to find the ...
1
vote
0answers
11 views

Sparse least squares where the coeefficient matrix is not stored explicitly

Consider a bounded linear operator $A : U \to V$ where $U$ is finite dimensional and where $V$ is a separable Hilbert space, or with large dimension such that $A$ cannot be stored explicitly, being ...
0
votes
0answers
21 views

Least square in dataset

I have a dataset with X and Y.I want to use the least square method to derive the correlation equation between X and Y. The question is, what dimension is appropriate for my equation? On what factors ...
0
votes
0answers
45 views

The derivative of the reciprocal of squared L2 norm

I want to solve part of an energy minimization problem like this: \begin{align*} \quad && \arg \min_{\textbf{X}} \sum_{i=1}^{I} \sum_{i' \in neighnors(i)} \frac{1}{ {\left\| (\textbf{x}^i - \...
15
votes
2answers
466 views

Least-squares solution to system of equations of $4 \times 4$ matrices with $2$ unknown matrices

This question is in the context of a robotics problem. The goal is to track a robot using both its onboard odometry system and a VR system (HTC Vive Pro) using a VR controller mounted to the robot. ...
1
vote
0answers
31 views

Least squares function approximation using Legendre polynomials

My task is to write a program that approximates a given function as a combination of $n$ first Legendre polynomials using Least Squares method in $[-1; 1]$. I understand that I need to minimize $$\...
0
votes
1answer
31 views

Least Squares - Why closed-form or LSMR solutions different from true $\beta$?

I wanted to try using the LSMR algorithm so I generated some data and run least squares. How come the LSMR solution and the closed-form one are different from the true $\beta$ I have used to generate ...
0
votes
1answer
28 views

Constrained Least Squares - Finding the Closest Solution to a Point

Below is the description of the problem that I'm stuck on: Suppose the wide matrix A has linearly independent rows. Find an expression for the point x that is closest to a given vector y (i.e., ...
0
votes
0answers
58 views

Orthogonal regressors in linear regression model

I am having trouble with proving one fact, which was left as an exercise on my statistics course: in linear regression model if $X \in \mathbb{R}^{n \times p}$ is regressors matrix and is orthogonal($...
0
votes
1answer
38 views

Why is this not a closed form solution to non-negative Least squares?

According to Solving Non Negative Constrained Least Squares by Analogy with Least Squares (MATLAB) and all resources I have looked up regarding this topic, the non-negative least squares problem $$ \...
0
votes
0answers
23 views

Is parameter identified in linear regression model?

One definition of identification of a parameter is that it can be uniquely expressed in terms of population moments of observed data. Consider the linear regression model $E[Y|X]=X'\beta,$ where $Y$ ...
1
vote
0answers
21 views

Is conditional expectation continuous in conditioning argument?

Suppose $X,Y$ are real-valued random vectors. When can we say $E[g(X,Y)|X]$ is an a.e. continuous function in $X$? I ask because I am reading Chapter 7 of Hansen, which asserts without proof that the ...
0
votes
0answers
32 views

Gradient of linear least squares problem as number of samples increases

Consider the following linear least squares problem with the objective of minimizing the loss function $$L = \frac{1}{N} \sum_{i=1}^N (\vec{c} \cdot \vec{x}_i - y_i)^2$$ where $N$ is the number of ...
1
vote
0answers
21 views

How bad is polynomial regression with OLS

Let us look at the following polynomial regression: $$ y_{i} = \beta_{0} + \beta_{1}x_{i} + \beta_{2}x_{i}^{2} + \dots + \beta_{m}x_{i}^{m} + \varepsilon_{i}. $$ Then, OLS solution is given by $$ \hat{...

1
2 3 4 5
33