Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere. See https://en.wikipedia.org/wiki/Least_squares

learn more… | top users | synonyms

0
votes
0answers
7 views

Identifying indeterminable terms in polynomial fit

I am using SVD to fit a polynomial surface to a set of points, where the number of points may be less than, equal to, or more than the number of polynomial terms. For simplicity, let's assume points ...
0
votes
0answers
15 views

Least square estimation (LSE) method to solve equations

I am trying to find out the disadvantages of using least square estimation to solve non linear equations.Kindly can some on please comment on this.Thanks in advance.
2
votes
1answer
42 views

Obtaining quadratic equation using Least Squares Method

This question is most likely extremely trivial, but I'm having some difficulty obtaining the least squares equation from the following data points: {{1.08, 0}, {1.07, 0.0659232}, {0.97, 0.1695168}, ...
0
votes
1answer
46 views

Solve $\min_{\mathbf{x}} \sum_i \min\left[ (\mathbf{c}_i^T\mathbf{x}-a_i)^2, (\mathbf{d}_i^T\mathbf{x}-b_i)^2 \right]$

I am wondering if there is an efficient (perhaps closed form) way to solve the following piecewise quadratic minimisation problem: $$ \min_{\mathbf{x}} \sum_{i=1}^n \min\left[ ...
0
votes
1answer
21 views

Gradient of cost function

I have tried to calculate the gradient of the LMS cost function as follows but have a problem. $$J(\theta) = \frac12(y - X'\theta)^2$$ where $y$ is a scalar, theta and $X$ is a $n$ dimensional vector ...
1
vote
0answers
10 views

Are ordinary least squares coefficients 'linear' in the following sense.

If I have two sets of noisy data, with same number of points in each set and measured at the same set of x positions, then carry out a polynomial least squares fits on each set of data, are the ...
0
votes
0answers
24 views

Using least squares regression to apply nonlinear function to time series data

If you have a nonlinear function (see example), can you use a least squares regression approach to fit it to time series data ? Is this approach also valid for n variables? How many time points are ...
2
votes
1answer
47 views

Estimating landing position for a slowly falling object using latitude, longitude, and altitude.

I have a weather balloon project, in which I intend to use GPS to locate the payload when it finally comes down again. I will make the computer send coordinates to a server every minute or so, as ...
2
votes
1answer
35 views

Fitting a curve given points

My set-up is the following, I have two variables $N$ and $TTR$, and I have these points for each variable: $N$ = [35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50] $TTR$ = [0.818, ...
0
votes
0answers
16 views

Effect of marginalization on Gauss-Newton equations

Consider the problem of minimizing the cost function $f(x)=\eta(x)^TW\eta(x)$, where $\eta(x)=z-h(x)$ is an error function between the observations (measurements $z$) and their prediction $h(x)$, and ...
1
vote
1answer
50 views

Multivariational regression

I have been given following model $\ln(y)=\beta_0+\beta_1a+\beta_2a^2+\beta_3a^3+\beta_4a^4+\beta_5b$ and a set of observations that describe relation between $y$ and $\{a, b\}$. The goal is to ...
2
votes
0answers
31 views

Least squares problem: am I solving it correctly?

So I have this question: In $\mathbb R^3$ with inner product calculate all the least square solutions, and choose the one with shorter length, of the system: $ x + y + z = 1 $ $ x + z = 0 $ ...
1
vote
1answer
33 views

Least-square-method statistic (EDIT IN MY LAST ANSWER)

Good evening, I have a problem with the least-square-method in statistic: I've looked for an example and I found this: In the book << Springer Series in Statistics >> by D. Brillinger, page ...
0
votes
0answers
13 views

Using sequence of least squares solutions to solve non-linear problems?

I do know about the iteratively reweighted least-squares and have played around with it to some success finding non-linear solutions (like minimizing non-2-norms to achieve solutions which seem to be ...
1
vote
2answers
39 views

linear least squares with equality constraints

I am looking for iterative procedures for solution of the linear least squares problems with equality constraints. That is, my problem is to solve $$\min_{x} \lVert{Ax-b} \rVert _2, \ ...
1
vote
1answer
38 views

Linear vs non-linear Least Squares

I am trying to understand the difference between linear and non-linear Least Squares. In the book I have it says: "If the parameters enter the model linearly then one obtains a linear LSP." "If the ...
0
votes
0answers
20 views

Non-sparse, Equality-only constrained least squares solving

I am attempting to implement an "equality constraints only" linear least squares problem solution in Python. Obviously I would like to use a good existing solver or matrix decomposition method ...
2
votes
0answers
47 views

Fitting a sequence of vector spaces

I have a sequence of $m$-dimensional vector spaces indexed by $t$ $$S_t = \text{span}\left(\vec{v}_{1,t},\vec{v}_{2,t},\ldots,\vec{v}_{m,t}\right)$$ All the $S_t$ are subspaces of an $n$-dimensional ...
0
votes
0answers
33 views

Way to verify a least-squares solution without actually solving for $x$ and $y$?

I just found the least squares solution of the system $\mathbf{x}A = \mathbf{b} = \begin{pmatrix} x & y \end{pmatrix}\begin{pmatrix} 3 & 2 & 1 \\ 2 & 3 & 2\end{pmatrix} = ...
1
vote
0answers
14 views

Multiplicative & additive measurement error models concerning logarithms

I understand that taking the logarithm of the multiplicative error model transforms it into the additive error model. Let $y'$ be the observed response variable, with $y$ being the true response ...
0
votes
0answers
20 views

Least square estimation using Euler discretization

I have a set of differential equations $\dot{x}_i = f_i(x, \theta)$, with $x = [x_1, \ldots, x_n]^\top \in \mathbb{R}^n$ and $\theta \in \mathbb{R}^m$. Measurements of the variables $x_i$ are ...
0
votes
1answer
33 views

QR Factorization for Inconsistent Linear System

I am trying to recreate the problem found here on finding the least squares solution to an inconsistent linear system via QR factorization. Can someone explain the part about adding on vectors so that ...
0
votes
2answers
30 views

How to formulate a data fitting problem as a least squares problem

Formulate the data fitting problem as a least squares problem $\frac {1}{2} \Vert Ax-b \Vert_2^2 $ I thought I was supposed to wrote it like this: $ \frac {1}{2} x^THx + g^T+ \gamma$ but actually ...
1
vote
1answer
21 views

Least squares optimization

In Least Square optimization, A=\begin{pmatrix} 1 & t_1 & t_1^2 & \cdots & t_1^k \\ 1 & t_2 & t_2^2 & \cdots & t_2^k \\ \vdots & \vdots& \vdots ...
0
votes
0answers
20 views

Error Covariance of Minimum-Variance Estimate

I'm working my way through Luenberger's "Optimization by Vector Space Methods". On chapter 4, "Least-Squares Estimation", Section 4.5., Theorem 1, Luenberger shows that given a measurement setup of ...
0
votes
1answer
29 views

Find least squares regression line

I have a problem where I need to find the least squares regression line. I have found $\beta_0$ and $\beta_1$ in the following equation $$y = \beta_0 + \beta_1 \cdot x + \epsilon$$ So I have both ...
1
vote
1answer
71 views

Finding the least-squares solution of Ax = b when the columns of A are orthonormal.

Find a formula for the least-squares solution of Ax = b when the columns of A are orthonormal.
0
votes
0answers
13 views

Nonlinear Optimization Residual Error Calculation with Sign Dependence

I'm working on some code to perform a simple nonlinear optimization. In this scenario, my objective function takes some number of inputs (maybe 3 to 6ish) and will return residuals (maybe 30 or 40). ...
0
votes
1answer
54 views

Show that $\hat x$ is a least squares solution of the system $Ax=b$

Show that if $\begin{bmatrix}A & I \\ O & A^T\end{bmatrix} \begin{bmatrix}\hat x \\ r\end{bmatrix} = \begin{bmatrix}b \\ 0\end{bmatrix}$ , then $\hat x$ is a least squares solution of the ...
1
vote
0answers
26 views

How to solve for best possible values for variables when more equations than variables?

I've been gathering a list of gun control laws that each state has, and I'm going to see which kinds of gun control laws work better than others. I'm sure that some will be somewhat effective, and ...
0
votes
2answers
21 views

Minimization using Singular value

Let $A$ be a $p\times q$ matrix, with rank $q$. Show that the vector $x$ that minimizes $\|Ax\|_2$ under the constraint $\|x\|_2 = 1$ is the right singular vector of $A$ corresponding to the smallest ...
0
votes
2answers
25 views

How to maximize matrix products

Find a unit vector v1 and a unit vector v2, such that the term: $$v^T \begin{bmatrix} 6 & -2 \\ -2 & 6 \end{bmatrix}v$$ is minimized and maximized, respectively. What are the minimum and ...
0
votes
2answers
30 views

The matrix equation for solving a straight line

So, I'm looking at this paper and trying to understand where equation 5 comes from. Looking at wikipedia, I see that they would use $\mathbf{X} = (\mathbf{A^T}\mathbf{A})^{-1}\mathbf{A^T}\mathbf{y}$ ...
0
votes
0answers
18 views

Analyticical solution of least square problem

could anyone explain: a) $||{Ax-b}||^2$ (there is also a lowered 2): what does this two 2's mean? b) why is the solution: $x =(A^TA)^{-1} A^Tb$ is? Thank you very much:)
0
votes
1answer
37 views

I think I'm making an error in my implementation of least squares but I'm not sure (python).

Starting to read a book and the author goes into least squares etc. and shows this pic. Thought I'd do it myself. Using the lin alg formula, there I noticed I was getting slightly different results, ...
1
vote
2answers
37 views

Confused regarding the interpretation of A in the least squares formula A^T A = A^T b

So I'm was watching gilbert strang's lecture to refresh my memory on least squares, and there's something that's confusing me (timestamp included). In the 2D case he has $A=[1,1;1,2;1,3]$. In the ...
1
vote
2answers
59 views

Unclear about matrix calculus in least squares regression

The loss function of a Least Squares Regression is defined as (for example, in this question) : $L(w) = (y - Xw)^T (y - Xw) = (y^T - w^TX^T)(y - Xw)$ Taking the derivatives of the loss w.r.t. the ...
0
votes
1answer
14 views

Least squares regression line $\hat{y_i}=a + b(x_i - \overline{x})$: proof $a = \overline{y}$

I was trying to follow the proof that shows that $a = \overline{y}$ in the least squares regression line $$\hat{y_i}=a + b(x_i - \overline{x})$$ but I don't understand why $\sum_{i=1}^{n} (x_i - ...
1
vote
1answer
30 views

Moore-Penrose inverse solves the Least square solution

The normal form $ (A'A)x = A'b$ gives a solution to the least square problem. When $A$ has full rank $x = (A'A)^{-1}A'b$ is the least square solution. How can we show that the moore-penrose solves ...
3
votes
1answer
106 views

Statistics, least square method

I am having problems with an exercise. I have some observations of the random variable $Y$: $0.17, 0.06, 1.76, 3.41, 11.68, 1.86, 1.27, 0.00, 0.04,$ and $2.10$. I know that $Y = X^2$ and that $X \sim ...
0
votes
1answer
41 views

Least squares and QR factorization

I have a full-column-rank matrix $A \in \mathbb{R}^{N \times n} $ ($N >> n$): $Q^{T} A = \begin{bmatrix} R & w \\ 0 & v \\ \end{bmatrix} , Q^{T} = \begin{bmatrix} c \\ d \\ ...
1
vote
1answer
34 views

Estimators in the case of refression with normally distributed errors

How can it be shown that the Maximum Likelihood Estimator and the Least Squares Estimator are equvalent in the case regression with normally distributed errors? Any help will be appreciated! Thanks in ...
0
votes
0answers
65 views

Solve the problem $Ax = 0$ when $A$ has full rank.

Generally, the answers $x$ of this least square problem $$Ax = 0$$ where $A = []_{m\times n}$ and $x = []_{n\times 1}$ are in the null space of $A$. I know that people usually use the right-most ...
0
votes
0answers
8 views

Is there any correlation between approximation trendline parameters?

Let's say I have two data sets $(x,y)$ and $(p,q)$ and two approximation trendlines: Logarithmic: $y = b·ln(x) + a$ Linear: $y = bx + a$ Let's say I applied logarithmic approximation to both data ...
1
vote
1answer
15 views

Does the correlation between the measurement noise influence the result of the LS?

Consider a linear measurement process with some noise: $$y=Hx+v$$ with $$v \sim \mathcal{N}(0,\Sigma)$$ the covariance matrix $\Sigma$ is not a diagonal matrix. As we know, using LS, the $\hat{x}$ is ...
0
votes
0answers
15 views

How to penalise unknowns in least-squares solution of system of linear equations

I have an equation of the form $Ax = b$ where $A$ has dimensions 87 by 66 and rank 60. The last 33 rows of my $A$ encode symmetries of the form $x_1 = x_2$. I know I can calculate a least-squares ...
0
votes
1answer
25 views

Standard deviation for measurements with errors - least squares?

I have been bugged by a simple problem in statistics recently. Let's assume that I have made a set of measurements of a certain quantity, each with an uncertainty estimate. I have a set of tuples ...
1
vote
0answers
43 views

Let $A$ be an 8 x 5 matrix of rank 3, and let $b$ be a nonzero vector in $N(A^T)$. Show $Ax=b$ must be inconsistent.

Here's the entire question: Let $A$ be an 8 x 5 matrix of rank 3, and let $b$ be a nonzero vector in $N(A^T)$. a) Show that the system $Ax = b$ must be inconsistent. Gonna take a wild stab at this ...
0
votes
1answer
32 views

Derive estimator for weighted linear regression

I can't figure out to derive estimator for normal equations for weighted linear regression. (Supposed to be similar to normal equations.) I set up problem as $W(y-XB)^T(y-xB)$ My Steps: $W(y^Ty - ...
0
votes
1answer
11 views

piecewise polynomial least squares fit

I have in mind a project involving a least-squares fit using piecewise polynomials; at a finite number of known arguments $x_j$, the $k_j$th derivative is discontinuous. How many basis functions are ...