Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Confidence Intervals from Nonlinear Least Squares (Numerical): How do I ensure reasonable values?

Summary: In numerical (solver-based) non-linear least squares, the smaller the step size scale (for finite differences), the smaller the 95% confidence intervals are. Ergo, by using a smaller step ...
0
votes
0answers
31 views

How to solve for BLUE in linear regression?

I have been studying OLS in matrix form and I understand that when $y=X\beta +\varepsilon$, and when $E[\varepsilon|X]=0$ and $Cov[\varepsilon|X]=\sigma^{2}I$, $\hat{\beta}=(X^TX)^{-1}X^{T}y$ is the ...
1
vote
0answers
17 views

How extreme singular values affects least square for Gaussian covariates.

Consider the generative model for linear regression w.r.t. the true parameter $w^* \in S^{d-1}$ $$y=Xw^*+e$$ with i.i.d. Gaussian error $e \sim N(0, \sigma^2I_n)$. Let $X \in \mathbb{R}^{n\times d}$ ...
0
votes
0answers
11 views

How to Reconstruct a curves from coordinate points that I extracted from edge detection technique?

I have extracted the coordinate points of a diagram using edge detection technique in opencv. I want to remap only those coordinate points that contribute to the curves in the diagram and make it look ...
0
votes
1answer
45 views

Sensitivity of least squares solution to a data point change

Consider computing the least squares solution in $x$: $$ \text{min}.~\|Ax - b\|^2, $$ when $A$ is an $m \times n$ matrix, and $b$ is a column vector. Let's suppose we now switch exactly one row of $A$...
0
votes
0answers
35 views

Fitting least squares to 3-D points to get an equation for a plane

Here is a thread that is inspiring this question. I don't agree with Ben's answer for the reason wcochran stated. I am trying to find the equation for a plane specifically using least squares. So say ...
1
vote
1answer
42 views

How to find plane equation from 8 3D points with least square method

I have been working on school projects to find a plane equation from 8 3D points. Normally from 3 points, we can create a plane equation but when we have a lot of points, we want to find a good ...
2
votes
0answers
37 views

Solve differential equation with a least square method

I'm trying to solve the following differential equation: $$\frac{d^2u}{dx^2}=\frac{du}{dx}*u+u^2+x$$ $$x \in \Omega=[0,1]$$ $$BCS:u|_{x=0}=1;\frac{du}{dx}|_{x=0}=1$$ Least square method is one of ...
0
votes
1answer
18 views

Invertible $X^TX$ - what happens when you clone rows of $X$?

My question is inspired by https://stats.stackexchange.com/questions/70899/what-correlation-makes-a-matrix-singular-and-what-are-implications-of-singularit, in particular ttnphns's answer where they ...
1
vote
0answers
10 views

Least Squares and producing chemistry substances - ask references

This question came to me after discuss if there are nice examples of applied linear algerba topics in chemistry. A chemical compound is often a bunch of less complex chemical compouds put together to ...
1
vote
1answer
26 views

How does $r$ for univariate regression relate to the general definition of $R^2$?

We know that the correlation coefficient in a univariate regression case between $x$ and $y$ is $$r = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^n (x - \bar{x})^2\sum_{i=1}^n (...
1
vote
1answer
2k views

Analytical solution of least square problem

could anyone explain: a) $||{Ax-b}||^2$ (there is also a lowered 2): what does this two 2's mean? b) why is the solution: $x =(A^TA)^{-1} A^Tb$ is? Thank you very much:)
4
votes
2answers
90 views

Linear Least Squares with Monotonicity Constraint

I'm interested in the multidimensional linear least squares problem: $$\min_{x}||Ax-b||^2$$ subject to a monotonicity constraint for $x$, meaning that the elements of $x$ are monotonically increasing: ...
1
vote
0answers
23 views

Alternative form for Fisher Matrix

I am trying to understand a line from a textbook which expresses the Fisher information matrix in an alternative form assuming Gaussian noise. In the case of Gaussian noise the log likelihood function ...
0
votes
0answers
10 views

How do you obtain the right Moore-Penrose inverse?

Let's say $M < N$ and $H$ is a $M \times N$ complex matrix, $W$ is a $N \times M$ complex matrix and $x$ is a M-dimensional complex vector. It is obvious that when $HWx = x$, W is given by the ...
1
vote
2answers
23 views

Least squares problem regarding distance between two vectors in $\mathbb{R}^3$

I'm solving an exercise problem and was facing some confusion regarding how to solve it. The problem is (roughly translated to English): Given the following: $$\mathbf{A} = \begin{bmatrix} 2 & 0 ...
0
votes
0answers
22 views

Issue in calculating Cramer-Rao Lower Bound from Fisher information matrices

I am having trouble understanding an apparent paradox in calculating the CRLBs from a Fisher information matrix. Lets say that I have some data which is fitted to a model of $$S=Aexp(-bt^2)$$ where S ...
0
votes
0answers
22 views

Statistical inference-wise, should I just worry about the standard error of the random part of my model?

I am modelling my real world system as: $$Y = f(x) + E$$ where $f$ is my deterministic model with some parameters $\theta_{1}$, $\theta_{2}$, ... and $E$ is the random part (stuff I can't explain, ...
0
votes
0answers
21 views

Use of least-squares minimization to solve noisy systems of linear equations

This question is somewhat connected to a previous one I posted two months ago concerning solving linear, over-determined systems of equations of the form $Ax = b$, where $A$ is a matrix of ...
14
votes
3answers
17k views

Derivative of squared Frobenius norm of a matrix

In linear regression, the loss function is expressed as $$\frac1N \left\|XW-Y\right\|_{\text{F}}^2$$ where $X, W, Y$ are matrices. Taking derivative w.r.t $W$ yields $$\frac 2N \, X^T(XW-Y)$$ Why ...
3
votes
0answers
58 views

Find two numbers to minimize the sum of squares

Suppose we are given a sequence of positive numbers $0<a_1<a_2< \cdots < a_n$. Step 1. Choose an integer $m$ where $m \in \{1,2,\cdots,n\}$. After choosing $m$, we divide our numbers into ...
1
vote
2answers
276 views

The mean squared error of Linear least squares solution

Suppose I have a linear system $y=Ax+\varepsilon$ where $y\in\mathbb R^n$, $x\in\mathbb R^m$ and $A\in\mathbb R^{n\times m}$. I have known that the least squares solution for $x$ is $\hat{x}=A^+y$ ...
1
vote
0answers
22 views

Minimum norm in least squares linear system

I realize this question is a bit unorthodox, but hopefully it's simple enough to understand. I want to solve the linear system $Ax = y$ where say $A \in \mathbb R^{n \times d}, x \in \mathbb R^d, y \...
0
votes
0answers
14 views

Does ALS require to fill the missing value as 0 first?

In many papers or posts, it is stated that ALS only minimizes the least square of observed ratings. $$L=\sum_{u, i \in S}\left(r_{u i}-\mathbf{x}_{u}^{\top} \cdot \mathbf{y}_{i}\right)^{2}+\lambda_{x} ...
0
votes
0answers
17 views

$\mathbb{R}^2\rightarrow\mathbb{R}$ bases with axisymmetric span

$\{(x\in\mathbb{R},y\in\mathbb{R})\mapsto x^iy^k|i,j\in\mathbb{N}_0,i+j\le n\}$ is for any $n\in\mathbb{N}$ a basis with span that is invariant under rotations in $(x,y)$. For example for $n=1$: A ...
0
votes
0answers
8 views

How to obtain inverse data from a given 3D polynomial fit coefficients

I have 3D curves where I fit 4th order polynomials. I use three sets of data XS, YS, and ZS in pairs, so I get coeffcients for ZS, XS; and ZS, YS separately (here the independent variable is Z). So, ...
0
votes
0answers
32 views

Any additional tips on curve fitting?

Suppose I have a model $$y(x) = F(x, a, b, c, d)~,$$ where the function $F$ is in general non-linear. I also have set of observations $(x_{obs}, y_{obs}, \Delta y_{obs})$. I would like to fit the ...
1
vote
1answer
21 views

$\|\hat{\beta} - \beta^*\|_2^2$ approximated in big O notation as $O(\|\hat{\beta}\|^2_2)$

I have a case where I have a linear regression model: $$y = X\beta^* + w$$ where $y \in \mathbb{R}^{n}$, $X \in \mathbb{R}^{n \times 1}$, $\beta^* \in \mathbb{R}^{1}$ and $w \sim N(0,I_n), w \in \...
3
votes
2answers
135 views

When solving a linear system, why SVD is preferred over QR to make the solution more stable?

I have seen many posts stating that SVD is more stable as a preprocessing for solving least square or linear system problem than QR. Certainly QR is less expensive than SVD, so I guess it makes sense....
0
votes
1answer
23 views

Question related to regression and interpolation

Suppose I have a non linear function between two variables $x(t)$ and $y(t)$ where $x$ and $y$ are both functions of some variable $t$. Now, i am primarily interested in the ratio of ${\frac {dx}{dt}}$...
0
votes
1answer
23 views

How to convert Linear Least squares to exponential model

I have data that is in an exponential form, however, I have been told to analyse it using least squares regression to get the equation y=mx+b before converting this to an exponential model. I was told ...
0
votes
1answer
44 views

Solving LMS system with trigonometric functions

I have a bunch of points $[x_n, y_n, z_n]$ and I want to find the angles $\vartheta_s$, $\vartheta_t$ and $\vartheta_y$ that minimize \begin{equation} \begin{split} &\sum_{n=0}^{N} (\, (\cos{\...
0
votes
1answer
8 views

Condition inequality in perturbed LS

I have two matrices $A \in \mathcal{M}_{n,d}(\mathbb{R})$ and $B \in \mathcal{M}_{d,d}(\mathbb{R})$ with $B$ being symmetric definite-positive. I am trying to find a condition on $A$ for which I have ...
-2
votes
2answers
31 views

Making a Negative Number Possible to Square Root

We are able to solve x^2+4=0 by square rooting both sides, but if we have x^2=-4 we can't solve. Firstly, why? Aren't they equal expressions? Secondly, if we have x^2=-4, why can't we bring the four ...
2
votes
2answers
202 views

Deriving the identity: $\hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}$

For some reason I am having an extremely hard time finding out how the following expression is derived $$ \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} $$ Is ...
0
votes
0answers
19 views

Solving nonlinear least-squares with first order Taylor expansion

I'm referencing this Wikipedia article. I understand that a Taylor expansion of a function $f(x)$ around $x = a$ can be given by $f(x) = \sum_{n = 0}^{\infty} \frac{f^{(n)}(a)}{n!}(x - a)^n$ ...
0
votes
0answers
17 views

Deriving OLS estimator dependent on X being orthogonal

Let $X \in \mathbb{R}^{n \times d}$ be a predictor matrix with orthonormal columns and $y \in \mathbb{R}^n$ as output vector and vector (OLS estimator) $\beta_{LS} \in \mathbb{R}^d$ for estimating a ...
0
votes
0answers
21 views

Ratio of norms as goodness of fit measure for least square

Let $x,y\in \mathbb{R}^n$ be two vectors. One way to think about a linear regression on $(x,y)$ is that there is a random variable $X$ and two real numbers $\beta_0$ and $\beta_1$ such that $Y = \...
2
votes
5answers
40 views

Why does adding $\lambda \boldsymbol{I}$ to $\boldsymbol{X}^T\boldsymbol{X}$ for $\lambda > 0$ guarantee invertibility?

This question is inspired by regularized least squares, where it is stated that $$ X^TX + \lambda I $$ is guaranteed to be invertible for all $\lambda > 0$. Is there an intuitive reason for how ...
1
vote
0answers
31 views

Why I am getting similar $\beta$ for minimization $\sum_i (\log(y_i)-X_i\beta)^2 $ and $\sum_i (y_iX_i\beta-e^{X_i\beta})$?

I asked a question here and learned that minimizing $\Vert \! \log(y)-X\beta \Vert_2^2$ and $\Vert y-e^{X\beta} \Vert_2^2$ are different. But I still have difficult times to understand why ...
1
vote
1answer
43 views

Is minimizing $\Vert \! \log(y)-X\beta \Vert_2^2$ and $\Vert y-e^{X\beta} \Vert_2^2$ the same?

I am trying to fit some exponential data ($y$ is the regression target vector, and $X$ is the data matrix, $\beta$ is the coefficients that we want to optimize). However I am getting different ...
0
votes
1answer
74 views

Least Squares: Derivation of Normal Equations with Chain Rule (Revisited)

My question pertains to someone else's answered question that has made me curious. The OP wanted to differentiate the following using the chain rule: $$ J(\theta)=\frac12(X\theta-y)^T(X\theta - y) $$ ...
1
vote
1answer
59 views

Minimizing $\| x A - B\|_F^2$ With a Constraint

I have previously asked an optimization question Here. I will reiterate the question and simply add a constraint to it: I have 2 known grayscale images (256×256 matrices) $A$ and $B$ and want to find ...
0
votes
0answers
56 views

Comparison of two least-squares optimization problems

I have come across two least square minimization problems. The first one is: $$\min_{\beta\in \mathbb{R}} \lvert y_j-x_j\beta\rvert, \quad \text{where}\ j = 1, \dots, n.$$ Here $y$ is the dependent ...
1
vote
0answers
16 views

Least absolute deviations problem minimization

Least absolute deviations problem minimization $\min_{β∈R}|y_j − x_{j1}β_1|$, for j = 1,....N Here y is the dependent variable and x is the independent variable. What happens to the case when $x_{...
0
votes
0answers
13 views

Covariance generated from best-fit chi error function

I came across definition of covariance matrix that is defined from best fit error equation. I would like to clarify correctnes of procedure. We define equation for error: \begin{equation}\label{eq:...
0
votes
0answers
20 views

Compare the least square estimators and the residuals of two Linear Regression with an alternative regressor

So I have been given this as an assignment for my econometrics course and I seriously can't understand where to begin here: Consider the least square regression $ y ∈ R^n $ on $ X ∈ R^{n*k} $, and the ...
0
votes
2answers
29 views

Calculating coefficients for least squares [cross-posted from CrossValidated]

In this blog, this author says to calculate the coefficients for the equation $$ Flat(x, y) = A + Bx + Cy + Dx^2 + Ey^2 + Fxy $$ using least squares. I found this PDF that shows how to do the ...
0
votes
0answers
37 views

Proof that least squares estimators are unbiased under gauss-markov assumptions

I saw two different derivations of $E[\hat{\beta}] = \beta$, and they don't appear to be equivalent to me. Method 1 (from https://www.youtube.com/watch?v=T5kjKqkCvHc). Method 2 (from https://www....
0
votes
0answers
10 views

How to increase speed by skipping calculations when fitting a curve with conjugate gradient?

Let us assume we have a least squares fitting problem. $${\bf v_o} = \min_{\bf v}\{\|{\bf \Phi v-d}\|_2^2\}$$ Where $${\bf \Phi} \in \mathbb R^{N\times k}\\{\bf d} \in \mathbb R^{N\times1}$$ ...

1
2 3 4 5
28