Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

Filter by
Sorted by
Tagged with
0
votes
1answer
7 views

Condition inequality in perturbed LS

I have two matrices $A \in \mathcal{M}_{n,d}(\mathbb{R})$ and $B \in \mathcal{M}_{d,d}(\mathbb{R})$ with $B$ being symmetric definite-positive. I am trying to find a condition on $A$ for which I have ...
-2
votes
2answers
31 views

Making a Negative Number Possible to Square Root

We are able to solve x^2+4=0 by square rooting both sides, but if we have x^2=-4 we can't solve. Firstly, why? Aren't they equal expressions? Secondly, if we have x^2=-4, why can't we bring the four ...
0
votes
0answers
18 views

Solving nonlinear least-squares with first order Taylor expansion

I'm referencing this Wikipedia article. I understand that a Taylor expansion of a function $f(x)$ around $x = a$ can be given by $f(x) = \sum_{n = 0}^{\infty} \frac{f^{(n)}(a)}{n!}(x - a)^n$ ...
0
votes
0answers
17 views

Deriving OLS estimator dependent on X being orthogonal

Let $X \in \mathbb{R}^{n \times d}$ be a predictor matrix with orthonormal columns and $y \in \mathbb{R}^n$ as output vector and vector (OLS estimator) $\beta_{LS} \in \mathbb{R}^d$ for estimating a ...
0
votes
0answers
20 views

Ratio of norms as goodness of fit measure for least square

Let $x,y\in \mathbb{R}^n$ be two vectors. One way to think about a linear regression on $(x,y)$ is that there is a random variable $X$ and two real numbers $\beta_0$ and $\beta_1$ such that $Y = \...
2
votes
5answers
37 views

Why does adding $\lambda \boldsymbol{I}$ to $\boldsymbol{X}^T\boldsymbol{X}$ for $\lambda > 0$ guarantee invertibility?

This question is inspired by regularized least squares, where it is stated that $$ X^TX + \lambda I $$ is guaranteed to be invertible for all $\lambda > 0$. Is there an intuitive reason for how ...
1
vote
0answers
31 views

Why I am getting similar $\beta$ for minimization $\sum_i (\log(y_i)-X_i\beta)^2 $ and $\sum_i (y_iX_i\beta-e^{X_i\beta})$?

I asked a question here and learned that minimizing $\Vert \! \log(y)-X\beta \Vert_2^2$ and $\Vert y-e^{X\beta} \Vert_2^2$ are different. But I still have difficult times to understand why ...
1
vote
1answer
42 views

Is minimizing $\Vert \! \log(y)-X\beta \Vert_2^2$ and $\Vert y-e^{X\beta} \Vert_2^2$ the same?

I am trying to fit some exponential data ($y$ is the regression target vector, and $X$ is the data matrix, $\beta$ is the coefficients that we want to optimize). However I am getting different ...
1
vote
0answers
15 views

Least absolute deviations problem minimization

Least absolute deviations problem minimization $\min_{β∈R}|y_j − x_{j1}β_1|$, for j = 1,....N Here y is the dependent variable and x is the independent variable. What happens to the case when $x_{...
0
votes
0answers
13 views

Covariance generated from best-fit chi error function

I came across definition of covariance matrix that is defined from best fit error equation. I would like to clarify correctnes of procedure. We define equation for error: \begin{equation}\label{eq:...
0
votes
0answers
19 views

Compare the least square estimators and the residuals of two Linear Regression with an alternative regressor

So I have been given this as an assignment for my econometrics course and I seriously can't understand where to begin here: Consider the least square regression $ y ∈ R^n $ on $ X ∈ R^{n*k} $, and the ...
0
votes
0answers
56 views

Comparison of two least-squares optimization problems

I have come across two least square minimization problems. The first one is: $$\min_{\beta\in \mathbb{R}} \lvert y_j-x_j\beta\rvert, \quad \text{where}\ j = 1, \dots, n.$$ Here $y$ is the dependent ...
0
votes
1answer
70 views

Least Squares: Derivation of Normal Equations with Chain Rule (Revisited)

My question pertains to someone else's answered question that has made me curious. The OP wanted to differentiate the following using the chain rule: $$ J(\theta)=\frac12(X\theta-y)^T(X\theta - y) $$ ...
0
votes
2answers
27 views

Calculating coefficients for least squares [cross-posted from CrossValidated]

In this blog, this author says to calculate the coefficients for the equation $$ Flat(x, y) = A + Bx + Cy + Dx^2 + Ey^2 + Fxy $$ using least squares. I found this PDF that shows how to do the ...
0
votes
0answers
37 views

Proof that least squares estimators are unbiased under gauss-markov assumptions

I saw two different derivations of $E[\hat{\beta}] = \beta$, and they don't appear to be equivalent to me. Method 1 (from https://www.youtube.com/watch?v=T5kjKqkCvHc). Method 2 (from https://www....
0
votes
0answers
10 views

How to increase speed by skipping calculations when fitting a curve with conjugate gradient?

Let us assume we have a least squares fitting problem. $${\bf v_o} = \min_{\bf v}\{\|{\bf \Phi v-d}\|_2^2\}$$ Where $${\bf \Phi} \in \mathbb R^{N\times k}\\{\bf d} \in \mathbb R^{N\times1}$$ ...
-1
votes
0answers
13 views

least squares: solve for b hat without knowing y

I am trying to solve for $\hat{B}$ without knowing the $y$ vector. I know $\hat{B}=((X'X)^{-1})X'y$ and i am given the norm of $y$, the projection of $y (X\hat{B})$, and the $X$ matrix (non-invertible)...
1
vote
1answer
58 views

Minimizing $\| x A - B\|_F^2$ With a Constraint

I have previously asked an optimization question Here. I will reiterate the question and simply add a constraint to it: I have 2 known grayscale images (256×256 matrices) $A$ and $B$ and want to find ...
0
votes
1answer
40 views

Find all solutions of least squares problem

I have the following exercise (this is exercise 4.39 of Fundamentals of Matrix Comuptations - Watkins) : I am not sure about how to find all the solutions(item e). I think I must use itens c) and d) ...
0
votes
2answers
39 views

How do I proof that $A=\sum\limits_{i=1}^{m}x_{i}x_{i}^{T} $ is invertible if and only if $X$ has full rank?

Show that $A=\sum\limits_{i=1}^mx_ix_i^T$ is invertible if and only if $x_1,\cdots,x_m$ span $\mathbb R^d$ for $x_i\in\mathbb R^d$. Here are my thoughts: If $A$ is invertible $Aw=0$ only has the ...
1
vote
1answer
40 views

Fundamentals of Matrix Computations, Watkins, exercise $4.3.9(e)$, SVD.

Given that $$A=\begin{bmatrix} 1 & 2 \\ 2 & 4 \\ 3 & 6\end{bmatrix}, \qquad b=\begin{bmatrix} 1 \\ 1\\ 1\end{bmatrix},$$ what is the method to find all solutions of the least-squares ...
1
vote
0answers
44 views

Find $a_j$ give m points as $(x_i,y_i)$ where: $i = 1,2,…,m$, $y=\sum_{j=0}^n a_j x^j \quad n < m$

I have been trying to solve this problem, is this answer right? it's my first time solving a problem in the curve fitting least-squares that has a finite series in it. find $a_j$ give m points as $(...
0
votes
1answer
23 views

What are the “moment conditions” in the GMM method? Also: GMM vs IV vs 2-stage least squares?

GMM = generalized method of moments IV = instrumental variables 2SLS = Two stage least squares OLS = ordinary least squares I keep seeing talk of 'moment conditions' or 'moment equations', but don'...
0
votes
0answers
27 views

Understanding Least Squares with Normal Equations

Recently in my lectures we did Householder reflectors and normal equations to solve $Ax = b$, with $A$ being a rectangular $m\times n$ matrix and $x$ being a $m\times 1$ vector, where $m>n$. Or ...
0
votes
0answers
14 views

Finding the coefficients error by the least squares method

How do I calculate the error $(\triangle) \space$of coefficients $a, b $ from $$ y = a + bx $$ if $$ \triangle y = \sqrt{\frac{1}{(n-2)} \sum_{i= 1}^n {(y_i - a - bx_i )^2}} $$ using the least ...
0
votes
0answers
21 views

Estimate a periodic time series using least square fit

Hi I was attempting the following question and I am stuck and I don't know how to proceed. From My understanding I first substituted $\hat{y} = Ax$ where A is T x P selector matrix , in the minimizing ...
0
votes
0answers
39 views

Sensitivity of least squares to the number of equations

I faced an interesting problem when studying linear squares method. Let there be an overdetermined system $$Ax=b$$ with, say, $A=[30\times 8]$. By solving the normalized system $A^TAx=A^Tb$ we get an ...
0
votes
0answers
7 views

Why are there differences in simple linear regression in machine learning as compared to when we were taught that previously in statistics?

I was taught previously in statistics class, that the equation for simple linear regression model : E(Y)=Beta0+beta1*x1 I recently took up a course on machine learning and in that the equation is ...
0
votes
0answers
36 views

Nonlinear least squares uniqueness

Suppose I have a nonlinear least squares objective function I want to minimize: $$ \chi^2(\mathbf{x}) = \sum_{i=1}^n f_i(\mathbf{x})^2 = \mathbf{f}(\mathbf{x})^T \mathbf{f}(\mathbf{x}) $$ Now suppose ...
0
votes
0answers
24 views

“Batchwise” least squares with smoothness in row direction as extra objective

My math background is essentially non-existant, so please bear with me. I have a "batchwise" (for lack of a better term) linear least squares problem $A X = Y$ that I solve like $\hat X = A^\dagger Y$...
0
votes
0answers
28 views

Lagrangian multipliers in complex least square

We need to solve the following complex least square problem:\begin{align*}\min\limits_X&&\|Y-X\|^2\\\text{s.t.}&&X^TAX=0\end{align*}where the complex matrix $Y$ and the real matrix $A$...
0
votes
0answers
3 views

How to prove independence based on condition for linear regression

Suppose that we have the following linear model $\mathbb{E}[Y_{i}]=b_{0}+b_{1}(x_{i}-\bar{x})$ where $i=1,...,n$ and $\bar{x}$ is the mean of $x_{i}$ and $Y_{i}$ are uncorrelated with constant ...
0
votes
0answers
18 views

Linear Algebra for Least Square Lines for Data

Find the least-squares line for the data below by following the steps after the data. Data Given The year 1880 1910 1940 1970 2000 Global Mean Temperature 13.73O ...
0
votes
1answer
50 views

A box-constrained least-squares problem

I need to solve the following problem in $\mathbf{x} \in \mathbb R^p$. $$\min_{\mathbf{s} \in [-1,1]^p}\left\Vert \mathbf{x}-\lambda\mathbf{s}\right\Vert _{2}$$ where $\mathbf{s} \in \mathbb R^p$ ...
1
vote
0answers
17 views

Solving least squares with QR

In order to avoid $A^TA$'s bad condition we decompose $A$ into $A = QR$ , $Q \in \mathbb{R}^{n*m} $ , $R \in \mathbb{R}^{m*m} $ , $Q^TQ = I$ . But in my textbook it says that $$ min||r||_2^2 = min||b -...
0
votes
0answers
20 views

Error in linear least squares

If I have an overdetermined system of linear equations $Ax=b$, I can solve by least squares. The error vector is: $e=Ax-b$, which is the error that is being minimised by the least squares method. ...
0
votes
1answer
8 views

Least Squares and Preconditioners

I've seen the preconditioned least squares objective function $\arg\min_{x}\Vert P^{-1}Ax-P^{-1}b\Vert_2$, where $P^{-1}$ is positive definite. However, I'm not sure how we show that this is the ...
0
votes
0answers
7 views

Typo in Wasserman explanation of Reweighted Least Squares?

My test (All of Statistics, second edition, Wasserman) contains the following: Reweighted Least Squares Algorithm (for Logistic Regression) Choose starting values $\hat{\beta}^0 = (\hat{\...
1
vote
0answers
19 views

Least Square with quadratic constraint

We need to solve the following least square problem \begin{align} Y=X\theta+W, \end{align} with quadratic constraint \begin{align} \theta^TA\theta = 0, \end{align} where the complex matrix $X \...
0
votes
0answers
26 views

Compute grid orientation, spacings, skew and origin from a set of 2D points

I have a collection of 2D points that are originated from a regular grid (rectangular grid most of the time, but possibly skewed as well). Each point can be located on the grid from its Row and Column ...
0
votes
0answers
16 views

What is the connection between MSE in regression and MSE for prob. distributions?

When assigning a goodness of fit to the least squares regression, one often naturally takes the mean squared error (MSE) or average residual: $$MSE = \frac{1}{n}\sum_{i}{(y_{i}-\hat{f}_{D}(x_{i}))^2}$$...
3
votes
2answers
44 views

linear least squares — complex observations, real estimate constraint

Consider the following least squares optimization problem: $$ \hat{x} = \arg\min_x \| y - A x\|^2 $$ where the observations are complex $y\in{\cal C}^{N\times 1}$, and the complex design matrix $A\...
3
votes
2answers
85 views

Deriving the Jacobian and Hessian of the nonlinear least-squares function

I'm working on a problem that involves deriving the Jacobian and Hessian of the following nonlinear least squares function. I've been thinking of expanding the function with Taylor Series expansion ...
0
votes
0answers
21 views

Least squares, what are the new points?

Let's say I have the following equation system: equation system The points are graphed as follows: graph of points Using the least squares method a solution is x =3 and y=1. According to the ...
2
votes
1answer
72 views

How to obtain the closed form expression of least-square sphere fitting?

I see here and there that there is a closed form expression of the least-square fit of a sphere of radius $r$ and center $\mathbf{c}$ to $N$ data points $\{\mathbf{x_i}\}_{i\in(1,\cdots,N)}$. How is ...
-1
votes
1answer
31 views

Optimization Kronecker

Now I have to solve a optimization problem \begin{equation} u=\sum_{p=1}^{P}h_{1,p}\otimes h_{2,p}.\\ \min_{h_{1,p},h_{2,p}}u^TRu, \end{equation} with a iterative algorithm, where R is a correlation ...
0
votes
0answers
20 views

Finding the best fit for the data

Hi guys i am trying to use the normal equations to fit some data to find values a and b for the following equation. $$y=a*b^x$$ The data was given to me below and this is what i am currently using ...
0
votes
2answers
49 views

Least squares and null space

I want to solve a least squares problem, $$ \min_x ||y - A x ||^2 $$ with $A \in \mathbb{R}^{m\times n}$. Suppose I were to find two distinct solutions $x_1,x_2$, which solve the problem, so that $$ ||...
0
votes
1answer
35 views

$\arg\min \|x-x_0\|²$ s.t. $Ax=b$

Let $A\in \mathbb R^{m×n}, b \in \mathbb R^m$ with $m \leq n$, $\operatorname{rank} A = m$ and $x_0 \in \mathbb R^n$. Consider the problem : $$\arg \min\|x-x_0\|² \quad\text{s.t. }Ax = b$$ How can I ...
0
votes
0answers
27 views

Pulling out terms in solving the covariance of the slope and intercept in linear regression

I want find the covariance of the estimates $\hat{\beta_0}$ and $\hat{\beta_1}$. There are many answers such as this one that give the answer as \begin{align*} \operatorname{Cov}(\hat{\beta_0}, \hat{\...

1
2 3 4 5
27