Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

1,642 questions
Filter by
Sorted by
Tagged with
5 views

How do you find the asymptotic distribution of the OLS estimator in a simple linear regression model with an intercept?

How do I find the asymptotic distribution of $\beta_1$ in the model $y_i=\beta_0+\beta_1x_i+\epsilon_i$? I am able to follow Cameron's derivation of a very similar model without an intercept, but when ...
12 views

Legacy code solving least squares adjustments

I was given the task to maintain an old library that, among other things, claims to calculate least squares adjustments. I have read and understood some theory behind it, however I still have troubles ...
24 views

Finding the right optimisation algorithm for nonlinear ARX Model

I am trying to explain my problem in a simplified way: (P1) In the past I have solved the following optimisation problem using the Levenberg–Marquardt algorithm (LVM) offline: Output: $v_e$ Input: i ...
20 views

Losing symmetry in least squares approximation

I tried to find the best linear fit to $(1,-1)$, $(-1,1)$, $(-2,-2)$, and $(2,2)$. There is symmetry within the four points: across $y=x$ and $y=-x$. However, the least squares linear approximation ...
17 views

Kronecker product identity when multiplied by two vectors?

I am interested in least-squares optimization for problems with space-time separable prior state covariances and am trying to break down the quadratic cost function into respective space-time ...
31 views

Optimality conditions of LASSO

In this paper, on page 1122, it states that the optimality conditions for the LASSO give $\hat{\beta} = n_{\lambda}(\hat{\beta} - X^T(X\hat{\beta} - y))$, where $n_\lambda$ is the soft-thresholding ...
63 views

Algorithm to find minimum of sum of squares

I need find the set of integers $X = \{x_0, x_1, ...,x_n\}$ that minimizes: $$\sum_{i=0}^n \left(x_i - T\frac{W_i}{P_i}\right)^2$$ where: $\sum_{i=0}^n x_iP_i < T$ $x_0, x_1, ...,x_n$ are integers ...
16 views

How can I derive OLS predicted error term $\hat{e}_i$ as a function of $e_i$?

First of all, I'd like to say that any kind of help would be really helpful, whether it's a hint or a good grad/undergrad book. Right now I'm working with Econometric Analysis of Cross Section and ...
45 views

Upper bound for the error of the gaussian quadrature$\int_{0}^{1} \log (1+\operatorname{sin} x) \mathrm{d} x$

I'm given the integral $\mathrm{I}=\int_{0}^{1} \log (1+\operatorname{sin} x) \mathrm{d} x$. Through the formula of Gaussian quadrature for 3 points, I can find an approximation to this integral. The ...
16 views

Question regarding Least Squares Algorithm residuals

I was reading about Least Squares and there was an example, however there was a part of the code which I did not understand why we do that step mathematically-wise. (The code was in Matlab). First of ...
25 views

Variable transformation for a multiple linear regression model

I can only transform C) in a way that leaves me with only constants for the parameters. In all other functions I either end up with $ln(\beta_1)$ or need to take the square root of $\beta$, etc. I'm ...
62 views

Gauss-Newton normal equations with norm of residual

The Wiki definition of Gauss-Newton has the following scalar cost function: $S({\boldsymbol {\beta }})=\sum _{i=1}^{m}r_{i}^{2}({\boldsymbol {\beta }}).$ where $r_i(\beta)$ are scalar ...
25 views

25 views

16 views

Optimization problem given weight matrix

Suppose there is a unit norm vector $x \in R^n$, and $y$ is a linear combination of elements of $x$, and let $y = \sum_{i=1}^n w_i^3 x_i$. Given the knowledge of $w_i$, $w = [w_1, ..., w_n]^T$, are we ...
53 views

Closed form solution of $\displaystyle\arg\min_{\alpha \in \Bbb R} \|X - \alpha Y\|_{\text F}^2$

Given $m \times n$ matrices $X$ and $Y$, I am interested in the following least-squares problem. $$\hat \alpha := \arg\min_{\alpha \in \Bbb R} \|X - \alpha Y\|_{\text F}^2$$ Is there any way to ...
69 views

Why doesn't the Gauss-Newton method diverge around the minimum?

I am having trouble visualizing the convergence of the Gauss-Newton method. Consider the simple function of $f(x) = x^2 + 1$. If I try to use the Gauss-Newton method to find the minimum of $f(x)^2$ (...
27 views

31 views

Least Squares - Why closed-form or LSMR solutions different from true $\beta$?

I wanted to try using the LSMR algorithm so I generated some data and run least squares. How come the LSMR solution and the closed-form one are different from the true $\beta$ I have used to generate ...
28 views

Constrained Least Squares - Finding the Closest Solution to a Point

Below is the description of the problem that I'm stuck on: Suppose the wide matrix A has linearly independent rows. Find an expression for the point x that is closest to a given vector y (i.e., ...
I am having trouble with proving one fact, which was left as an exercise on my statistics course: in linear regression model if $X \in \mathbb{R}^{n \times p}$ is regressors matrix and is orthogonal(\$...