Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

Filter by
Sorted by
Tagged with
0
votes
0answers
37 views

Proof that least squares estimators are unbiased under gauss-markov assumptions

I saw two different derivations of $E[\hat{\beta}] = \beta$, and they don't appear to be equivalent to me. Method 1 (from https://www.youtube.com/watch?v=T5kjKqkCvHc). Method 2 (from https://www....
0
votes
0answers
10 views

How to increase speed by skipping calculations when fitting a curve with conjugate gradient?

Let us assume we have a least squares fitting problem. $${\bf v_o} = \min_{\bf v}\{\|{\bf \Phi v-d}\|_2^2\}$$ Where $${\bf \Phi} \in \mathbb R^{N\times k}\\{\bf d} \in \mathbb R^{N\times1}$$ ...
1
vote
1answer
59 views

Minimizing $\| x A - B\|_F^2$ With a Constraint

I have previously asked an optimization question Here. I will reiterate the question and simply add a constraint to it: I have 2 known grayscale images (256×256 matrices) $A$ and $B$ and want to find ...
0
votes
1answer
41 views

Find all solutions of least squares problem

I have the following exercise (this is exercise 4.39 of Fundamentals of Matrix Comuptations - Watkins) : I am not sure about how to find all the solutions(item e). I think I must use itens c) and d) ...
0
votes
2answers
39 views

How do I proof that $A=\sum\limits_{i=1}^{m}x_{i}x_{i}^{T} $ is invertible if and only if $X$ has full rank?

Show that $A=\sum\limits_{i=1}^mx_ix_i^T$ is invertible if and only if $x_1,\cdots,x_m$ span $\mathbb R^d$ for $x_i\in\mathbb R^d$. Here are my thoughts: If $A$ is invertible $Aw=0$ only has the ...
1
vote
1answer
42 views

Fundamentals of Matrix Computations, Watkins, exercise $4.3.9(e)$, SVD.

Given that $$A=\begin{bmatrix} 1 & 2 \\ 2 & 4 \\ 3 & 6\end{bmatrix}, \qquad b=\begin{bmatrix} 1 \\ 1\\ 1\end{bmatrix},$$ what is the method to find all solutions of the least-squares ...
1
vote
0answers
44 views

Find $a_j$ give m points as $(x_i,y_i)$ where: $i = 1,2,…,m$, $y=\sum_{j=0}^n a_j x^j \quad n < m$

I have been trying to solve this problem, is this answer right? it's my first time solving a problem in the curve fitting least-squares that has a finite series in it. find $a_j$ give m points as $(...
0
votes
1answer
29 views

What are the “moment conditions” in the GMM method? Also: GMM vs IV vs 2-stage least squares?

GMM = generalized method of moments IV = instrumental variables 2SLS = Two stage least squares OLS = ordinary least squares I keep seeing talk of 'moment conditions' or 'moment equations', but don'...
0
votes
0answers
27 views

Understanding Least Squares with Normal Equations

Recently in my lectures we did Householder reflectors and normal equations to solve $Ax = b$, with $A$ being a rectangular $m\times n$ matrix and $x$ being a $m\times 1$ vector, where $m>n$. Or ...
0
votes
0answers
14 views

Finding the coefficients error by the least squares method

How do I calculate the error $(\triangle) \space$of coefficients $a, b $ from $$ y = a + bx $$ if $$ \triangle y = \sqrt{\frac{1}{(n-2)} \sum_{i= 1}^n {(y_i - a - bx_i )^2}} $$ using the least ...
0
votes
0answers
21 views

Estimate a periodic time series using least square fit

Hi I was attempting the following question and I am stuck and I don't know how to proceed. From My understanding I first substituted $\hat{y} = Ax$ where A is T x P selector matrix , in the minimizing ...
0
votes
0answers
39 views

Sensitivity of least squares to the number of equations

I faced an interesting problem when studying linear squares method. Let there be an overdetermined system $$Ax=b$$ with, say, $A=[30\times 8]$. By solving the normalized system $A^TAx=A^Tb$ we get an ...
0
votes
0answers
7 views

Why are there differences in simple linear regression in machine learning as compared to when we were taught that previously in statistics?

I was taught previously in statistics class, that the equation for simple linear regression model : E(Y)=Beta0+beta1*x1 I recently took up a course on machine learning and in that the equation is ...
0
votes
0answers
40 views

Nonlinear least squares uniqueness

Suppose I have a nonlinear least squares objective function I want to minimize: $$ \chi^2(\mathbf{x}) = \sum_{i=1}^n f_i(\mathbf{x})^2 = \mathbf{f}(\mathbf{x})^T \mathbf{f}(\mathbf{x}) $$ Now suppose ...
0
votes
0answers
25 views

“Batchwise” least squares with smoothness in row direction as extra objective

My math background is essentially non-existant, so please bear with me. I have a "batchwise" (for lack of a better term) linear least squares problem $A X = Y$ that I solve like $\hat X = A^\dagger Y$...
0
votes
0answers
33 views

Lagrangian multipliers in complex least square

We need to solve the following complex least square problem:\begin{align*}\min\limits_X&&\|Y-X\|^2\\\text{s.t.}&&X^TAX=0\end{align*}where the complex matrix $Y$ and the real matrix $A$...
0
votes
0answers
3 views

How to prove independence based on condition for linear regression

Suppose that we have the following linear model $\mathbb{E}[Y_{i}]=b_{0}+b_{1}(x_{i}-\bar{x})$ where $i=1,...,n$ and $\bar{x}$ is the mean of $x_{i}$ and $Y_{i}$ are uncorrelated with constant ...
0
votes
0answers
18 views

Linear Algebra for Least Square Lines for Data

Find the least-squares line for the data below by following the steps after the data. Data Given The year 1880 1910 1940 1970 2000 Global Mean Temperature 13.73O ...
0
votes
1answer
59 views

A box-constrained least-squares problem

I need to solve the following problem in $\mathbf{x} \in \mathbb R^p$. $$\min_{\mathbf{s} \in [-1,1]^p}\left\Vert \mathbf{x}-\lambda\mathbf{s}\right\Vert _{2}$$ where $\mathbf{s} \in \mathbb R^p$ ...
1
vote
0answers
18 views

Solving least squares with QR

In order to avoid $A^TA$'s bad condition we decompose $A$ into $A = QR$ , $Q \in \mathbb{R}^{n*m} $ , $R \in \mathbb{R}^{m*m} $ , $Q^TQ = I$ . But in my textbook it says that $$ min||r||_2^2 = min||b -...
0
votes
0answers
20 views

Error in linear least squares

If I have an overdetermined system of linear equations $Ax=b$, I can solve by least squares. The error vector is: $e=Ax-b$, which is the error that is being minimised by the least squares method. ...
0
votes
1answer
8 views

Least Squares and Preconditioners

I've seen the preconditioned least squares objective function $\arg\min_{x}\Vert P^{-1}Ax-P^{-1}b\Vert_2$, where $P^{-1}$ is positive definite. However, I'm not sure how we show that this is the ...
0
votes
0answers
8 views

Typo in Wasserman explanation of Reweighted Least Squares?

My test (All of Statistics, second edition, Wasserman) contains the following: Reweighted Least Squares Algorithm (for Logistic Regression) Choose starting values $\hat{\beta}^0 = (\hat{\...
1
vote
0answers
20 views

Least Square with quadratic constraint

We need to solve the following least square problem \begin{align} Y=X\theta+W, \end{align} with quadratic constraint \begin{align} \theta^TA\theta = 0, \end{align} where the complex matrix $X \...
0
votes
0answers
29 views

Compute grid orientation, spacings, skew and origin from a set of 2D points

I have a collection of 2D points that are originated from a regular grid (rectangular grid most of the time, but possibly skewed as well). Each point can be located on the grid from its Row and Column ...
0
votes
0answers
16 views

What is the connection between MSE in regression and MSE for prob. distributions?

When assigning a goodness of fit to the least squares regression, one often naturally takes the mean squared error (MSE) or average residual: $$MSE = \frac{1}{n}\sum_{i}{(y_{i}-\hat{f}_{D}(x_{i}))^2}$$...
3
votes
2answers
49 views

linear least squares — complex observations, real estimate constraint

Consider the following least squares optimization problem: $$ \hat{x} = \arg\min_x \| y - A x\|^2 $$ where the observations are complex $y\in{\cal C}^{N\times 1}$, and the complex design matrix $A\...
3
votes
2answers
93 views

Deriving the Jacobian and Hessian of the nonlinear least-squares function

I'm working on a problem that involves deriving the Jacobian and Hessian of the following nonlinear least squares function. I've been thinking of expanding the function with Taylor Series expansion ...
0
votes
0answers
22 views

Least squares, what are the new points?

Let's say I have the following equation system: equation system The points are graphed as follows: graph of points Using the least squares method a solution is x =3 and y=1. According to the ...
2
votes
1answer
77 views

How to obtain the closed form expression of least-square sphere fitting?

I see here and there that there is a closed form expression of the least-square fit of a sphere of radius $r$ and center $\mathbf{c}$ to $N$ data points $\{\mathbf{x_i}\}_{i\in(1,\cdots,N)}$. How is ...
-1
votes
1answer
31 views

Optimization Kronecker

Now I have to solve a optimization problem \begin{equation} u=\sum_{p=1}^{P}h_{1,p}\otimes h_{2,p}.\\ \min_{h_{1,p},h_{2,p}}u^TRu, \end{equation} with a iterative algorithm, where R is a correlation ...
0
votes
0answers
20 views

Finding the best fit for the data

Hi guys i am trying to use the normal equations to fit some data to find values a and b for the following equation. $$y=a*b^x$$ The data was given to me below and this is what i am currently using My ...
0
votes
2answers
55 views

Least squares and null space

I want to solve a least squares problem, $$ \min_x ||y - A x ||^2 $$ with $A \in \mathbb{R}^{m\times n}$. Suppose I were to find two distinct solutions $x_1,x_2$, which solve the problem, so that $$ ||...
0
votes
1answer
36 views

$\arg\min \|x-x_0\|²$ s.t. $Ax=b$

Let $A\in \mathbb R^{m×n}, b \in \mathbb R^m$ with $m \leq n$, $\operatorname{rank} A = m$ and $x_0 \in \mathbb R^n$. Consider the problem : $$\arg \min\|x-x_0\|² \quad\text{s.t. }Ax = b$$ How can I ...
0
votes
0answers
27 views

Pulling out terms in solving the covariance of the slope and intercept in linear regression

I want find the covariance of the estimates $\hat{\beta_0}$ and $\hat{\beta_1}$. There are many answers such as this one that give the answer as \begin{align*} \operatorname{Cov}(\hat{\beta_0}, \hat{\...
0
votes
1answer
44 views

Solving LMS system with trigonometric functions

I have a bunch of points $[x_n, y_n, z_n]$ and I want to find the angles $\vartheta_s$, $\vartheta_t$ and $\vartheta_y$ that minimize \begin{equation} \begin{split} &\sum_{n=0}^{N} (\, (\cos{\...
0
votes
0answers
19 views

Compare matrices in the form of x^tAx

Hi I wish to compare 2 products of matrices like this : $(1 x_0 x_0^2) (X^TX)^{-1} (1 x_0 x_0^2)^T$ with $(1 x_0) (X_1^TX_1)^{-1} (1 x_0)^T$. Here X is the n by 3 design matrix and X1 is n by 2 ...
0
votes
0answers
61 views

Linear Least-Squares Frequency Domain

I am doing an implementation of the Poly-Reference Least Squares Complex Frequency Domain algorithm for modal analysis as described in various papers like: "A poly-reference implementation of the ...
0
votes
0answers
11 views

compare variances of ols regression function

Let $\tilde{f}$ denote the ols estimation of $f(x) = \beta_0 + \beta_1x_i$. Let $\hat{f}$ denote the ols estimation of $f(x) = \beta_0+\beta_1x_i+\beta_2x_i^2$. I wish to compare the variances of $\...
0
votes
1answer
22 views

Prove that there are unique values $w,b$ such that $L_2(w,b) = \sum_{i=1}^{n}(wx_i + b - y_i)^2$ is minimized.

I would like to prove following result: Suppose we have pair of points $D = \{(x_1,x_1),\cdots,(x_n,y_n)\}$. At least two of these points do not overlap (meaning that there is at least one pair of ...
0
votes
1answer
55 views

Curve fitting exponential function

in my schoolwork mathematical exploration, I am struggling to find ways in which I can model a function after a set of data. From a simulation software, I have gathered data that resembles an ...
0
votes
1answer
36 views

How to write $(AX-B)^{T}(AX-B)$ into the form $(X-K)^{T}\Sigma(X-K)$?

I have a problem in the derivation of matrix. Suppose $A$ is some $m \times n$ matrix, with $m>n$. $B$ is a $m \times 1$ vector. $X$ is a $n \times 1$ vector. If we define $M=(A^{T}A)^{-1}A^{T}$,...
1
vote
1answer
28 views

Asymptotic distribution of OLS estimator in a linear regression

I know the standard way of finding the asymptotic distribution of the OLS estimator in a linear regression. Suppose $$ y_i = x_i'\beta + u_i$$ where $u_i |X = x_i\sim N(0,\sigma^2)$. Let $\hat{\...
0
votes
0answers
21 views

Fit a plane to a set of 3D points by not using explicit function

I have a set of 3D points $\mathcal{X} = \left\{x_1,\ldots,x_N \right\}, x_i = (x_i^1, x_i^2, x_i^3)^T \in \mathbb{R}^3$ and I want to find a plane $c_0 x^1 + c_1 y^2 + c_2 x^3 + c_3 = c^T x = 0$ ...
0
votes
0answers
14 views

Perturbation analysis and sensitivity of eigenvector matrix product with specific perturbation

In my research in applied linear algebra and probability (Wiener filtering) I have come across this rather interesting problem: For a matrix $ U $ we denote by $ U_k $ the matrix formed by taking ...
-1
votes
1answer
42 views

Why Not Least Quartic Instead of Least Squares?

We are currently using Least Squares to calculate the error: $$\min_{a,b}\sum_{k=1}^n(ax_k+b-y_k)^2$$ Last squares magnifies the error making it bigger for larger errors, and this magnification ...
2
votes
2answers
27 views

Proof of Equivalence of Total Leat Squares Problem

Let $b \in \mathbb R^n $ and $A \in \mathbb R^{m\times n}$ with $m \geqslant n$ and $\operatorname{rank}(A)\le n$. Prove that the following statements are equivalent; $\hat{x} = \operatorname{argmin}...
0
votes
3answers
50 views

Plotting a parabola based on data points

I am trying to draw a parabola inside a chart which I am developing using D3.Js library and using SVG paths to draw the curve. I have a set of 5 points for drawing the parabola: ...
3
votes
2answers
55 views

Why in Statistics do we use R-squared when Comparing Linear Models instead of Least Squares?

In Machine Learning we use a cost function such as least squared errors to evaluate how good the model is and if one model has a better score than the other, assuming that it does not overfit we ...
0
votes
1answer
35 views

Can anyone advise me on how to solve this minimisation problem?

I have the following system of 3 equations and 3 unknowns: $$c_{0} = \frac{x_0}{x_0 + x_1},\ \ c_{1} = \frac{x_1}{x_1 + x_2},\ \ \ c_{2} = \frac{x_2}{x_2 + x_0},$$ where $c_i\!\in\!(0,1)$ are known ...

1
2
3 4 5
28