Questions tagged [least-squares]
Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.
1,833
questions
0
votes
0
answers
10
views
OLS assumption, full rank of matrix X
One of the OLS assumptions concerning the X-matrix (with a constant) is that the columns $(1, x_{i1}, · · · , x_{iK})$ are not linearly dependent. This looks intuitive to me, because of the dummy-...
0
votes
0
answers
16
views
Why is the norm the last $m - n$ elements of $Q^Tb$ in least squares?
The answer to this question says that
If you're solving the Least Squares problem minimizing $||Ax - b||_2$ then the error, or residual, is the norm of the last m-n elements of the vector $Q^Tb$.
...
- 389
1
vote
1
answer
44
views
Matrix equation solving involving vectorization and Kronecker product
I am looking an analytical solution for a least-squares solution of the following equation for $X$:
$(A \ (I_{J} \otimes X)) \ \textrm{vec}(X) = b, $
where $A \in \mathbb{R}^{I\times J^2}$, $I_J$ is ...
- 21
1
vote
1
answer
50
views
Can I use the least square fitting to obtain a real solution of a complex equation?
I am trying to solve the following equation:
$$\omega^{1/3} = p_l \omega_l^{1/3} + (1-p_l)\omega_{\omega}^{1/3} \tag{1}$$
$\omega$, $\omega_l$, and $\omega_w$ are complex numbers and $p_l$ must be ...
- 173
0
votes
0
answers
25
views
Least squares problem with second derivative regularization.
I want to minimise Least squares problem with second derivative regularisation, that is, minimizing the following w.r.t $\mathbf{g}(x)$.
$$min_{\mathbf{g}(x)} ||\mathbf{A} \cdot \mathbf{g}(x) - \...
-1
votes
0
answers
37
views
Function fitting curve
I have $(x_1,y_1),\dots (n_n,y_n)$ experimental data. I know that the function for fitting data is
$$y=21-A\left(1-\text{exp}\left(-(B+Cx)^D\right)\right)$$
Of course, I can not use Linear Regression. ...
- 792
2
votes
0
answers
35
views
Derivation of least squares solution using SVD for Lee-Carter mortality model
The Lee-carter model aims to model central mortality rates using the following
$$
log(m_{x,t}) = a_{x} + b_{x}k_{t} + \epsilon_{x,t}
$$
where $\epsilon_{x,t} \sim N(0,\sigma^{2})$
The following ...
- 21
0
votes
0
answers
21
views
Solving for constant of a known nonlinear dynamic system, numerically
Given a dynamic system,
\begin{equation*}
\frac{d\vec{s}}{dt}=f(t,\vec{s},\vec{c})
\end{equation*}
where $\vec{s}$ are the state, and $\vec{c}$ are constants.
And $\vec{s_0},\vec{s_1},...\vec{s_n}$ ...
- 1
5
votes
2
answers
135
views
How do I minimize a function where some of the variables are dependent on other variables?
Consider a camera placed in $(XC,YC,ZC)$.
The camera is looking down on a flat table $Z=0$:
Introduce a local coordinate system for the camera: $(x,y,z)$.
Initially the axes of the local coordinate ...
- 759
0
votes
0
answers
21
views
Mapping between LSQ for several samples and one single sample
I have a sysid problem of the type:
$Y = AX$,
where X is a matrix whose columns are different input vectors, and Y is the output.
Therefore, A is a general matrix that maps all the input vectors(...
- 59
1
vote
1
answer
60
views
Characterize the minimizers x*
While solving for least square case,
$\sum_{i=1}^m$$($$\sum_{j=1}^n$ $A_{i,j}$$x_{j}$ $-$ $y_{i}$)$^2$
the minimum value of $x$ in $||Ax-y||^2$ is characterized by the normal equation $A^TAx^*$ $=$ $A^...
0
votes
0
answers
47
views
what is the gradient of $||Ax -b||^2$?
I was trying to figure out the gradient descent approach of solving Ax-b for x. I have previously seen the closed form solution for x by minimizing $||Ax - b||^2$;
That gives the equation $x = (A^T A)^...
- 101
0
votes
0
answers
15
views
Seeking Review and Correction for Linear Equation and Least Squares Fit
I would greatly appreciate it if someone could review and correct my solution for the following problem involving a linear equation and least squares fitting. My requirement for solutions is that I ...
- 335
0
votes
1
answer
32
views
Approximate exponential-looking function
I'm having trouble with the following: I have made the following development graph ('total') based on some data from my job.
I would like to find a function that approximates this graph. The idea ...
- 603
1
vote
0
answers
22
views
Reformulation of a parameter estimation problem to use Least Squares Method
the measurement equation of the $i$-th sample is:
$0=r_i + 2 x_i^T P \dot x_i$, where $r_i\in\mathbb{R}_+, 0 \prec P\in\mathbb{R}^{n\times n}, x_i\in\mathbb{R}^n,\dot x_i \in\mathbb{R}^{n}$.
My goal ...
- 83
0
votes
0
answers
21
views
regularization and punishment method in least square fitting
I have a least square problem:
$$
\min \sum_j^p| f(x_j)-y_j |^2 \\
\text{where } f(x) = a x - \sum_{i}^{n} b_i J_1(c_i x)
$$
where $J_1$ is the first order Bessel function. I have to find a set ...
- 93
0
votes
0
answers
20
views
How to transform optimization problems involving Bessel functions into convex optimization problems
I have a set of data points $\{ (x_i,y_i) \}$. The target is to find a curve that fits these points best, so I use the least square method:
$$
\min \sum_j^p| f(x_j)-y_j |^2 \\
\text{where } f(x) = a x ...
- 93
0
votes
0
answers
13
views
Derivation of Feasible Generalized Least Squares Process (Wooldridge 2013)
I'm using Wooldridge's Introductory Econometrics (2013) and having some trouble understanding his derivation of the feasible generalized least squares (FGLS) method. He writes, assume that the ...
- 111
0
votes
1
answer
33
views
Approximating Linear Equations from Measurements
I've been working on a fascinating problem involving measurements that reveal a relationship between two variables, $t$ and $y$. I have a set of four data points: $(-1, 0)$, $(0, 1)$, $(1, 2)$, and $(...
- 335
0
votes
1
answer
34
views
Error vs residual in simple linear regression
In my textbook the following definition were presented
However, it was presented previously that
Thus if we were to minimise the sum of squared residuals, shouldn't we be minimising $Q=\sum_{i=1}^n(...
- 162
0
votes
0
answers
11
views
Rigid transformation and similarity transformation in MLS
I am reading the paper Image Deformation Using Moving Least Squares of Schaefer. I attach the link below:
https://people.engr.tamu.edu/schaefer/research/mls.pdf
In Section 2.3 Rigid Deformations, I am ...
- 1
-1
votes
1
answer
43
views
Best approximation with unknown basis functions
Given an analytic real known function such as $f(x)=\exp(-x^2)$ defined on a finite interval such as $x \in [0,1]$, find the solution of the following problem
$$\min_{c_k,\phi_k(x)} \int_a^b|e(x)|^2 ...
1
vote
0
answers
19
views
Optimal Choice of Regression points for Minimizing the Approximation Error when solving a PDE with Function Approximation
I want to solve a high-dimensional PDE $F(\mathbf{x}, v ,\triangledown v , \triangledown^2 v )$
$$
-\dot{V_{t}}(\mathbf{x}) - A (\triangledown v_t (\mathbf{x})) = f(\mathbf{x}) , \quad \mathbf{x} \...
- 41
0
votes
0
answers
24
views
Condition number for non-negative least squares
If I want to analyze the stability of a non-negative least squares problem $||Ax-b||_2^2, x\ge0$, how can I measure the stability of the system? If it was a regular least squares problem, then I can ...
- 101
0
votes
0
answers
22
views
Uniqueness of non-negative least squares solution
When solving linear problems, there are many different ways of assessing if $Ax=b$ has a unique solution for a given $A$ by looking at properties of $A$ like the rank or singular values. If I am ...
- 101
0
votes
0
answers
68
views
Prove Convergence of Iterative Least Square Algorithm in space of Basis Functions
I have the following problem where I have already established convergence in the general case. I would like to show the convergence in a lower-dimensional space spanned by basis functions.
General ...
- 41
0
votes
0
answers
29
views
Weighted Least squares with Multiple Unknowns and Iterations
I am currently working on a problem involving the minimization of the $\chi^2$ deviation between a model matrix ($C_{model}$) and a measured matrix ($C_{measured}$). by finding the best fit parameters ...
-1
votes
1
answer
27
views
How to calculate t-statistics in generalised least squares? [closed]
As the question states, how would you calculate the t-statistics in GLS? Is it just the same as in OLS? And are these results still reliable?
I have heteroskedastic data and so am using GLS, but I'm ...
0
votes
0
answers
11
views
How to calculate sample standard deviation of the intercepts that are obtained from all possible combinations (having $n \ge 3$ points)?
Starting with an example:
Having the points $P_1(1,29),P_2(2,50),P_3(4,88)$. Then there are $4$ possible ways to choose points to find the "best" line (using ordinary least-squares). The $4$ ...
- 4,800
0
votes
1
answer
54
views
Best estimator for geometric distribution
I need an estimator for geometric distribution $\text{Geom}(p)$ that best fits my data $X_1, X_2,\ldots$
Is $\widehat{p} = \dfrac{1}{\overline{X}}$ the answer? Both MLE and method of moments yield ...
- 1
1
vote
0
answers
35
views
Spot redundant equations within nonlinear system of equations.
Is there a general procedure to detect if in a system of m-nonlinear equations (also non polynomial) of n-unknowns some of the equations are redundant?
Can the rank of the Jacobian matrix tell me ...
- 73
1
vote
1
answer
79
views
How can we gain a conceptual understanding of why $A^TAx=A^Tb$ gives us the least squares approx. for $x$ in Linear Algebra? [closed]
Per the title, I am a visual learner and would appreciate some conceptual explanation as for why that equation works and is equivalent to $Ax = b$ for the least squares approximation.
If the answer ...
- 11
2
votes
1
answer
68
views
Least Squares with Inequality Constraints [closed]
I want to use Least Squares to minimize $Ax-b$ (overdetermined system), subject to $x_1+x_2+x_3=1$ and $\forall x_i, 0 \leq x_i \leq 1$. As per my understanding, I need to set up the Lagrange function ...
- 81
0
votes
0
answers
32
views
Minimizing the norm of a vector which is squared elementwise
I've encountered the following optimization problem, amazingly in a real-world engineering application:
$\min ||\omega^2||_2$
$s.t. H\omega=M$
where $\omega^2=[\omega_1^2\quad\omega_2^2\quad\omega_3^2\...
- 173
1
vote
0
answers
24
views
Multi-class least sqares with a single coefficient per intput dimension
Suppose we are given multi-class labels $\mathbf{y} \in \mathbb{R}^K$ where $K \in \mathbb{N^+}$ and data $\mathbf{X} \in \mathbb{R}^{P \times K}$ where $P \in \mathbb{N^+}$. We wish to minimise the ...
- 232
0
votes
0
answers
41
views
Understanding the implications of singular matrices
Context
An over-determined system of equations can not be generally solved by elimination operated on $Ax = b$.
Instead, it normally involves solving $$(A^t A) x = A^t b$$
which is obtained in the OLS ...
- 111
2
votes
1
answer
36
views
Polynomial interpolation but with a perfect fit on certain data points
When given $n$ data points $(x_1,y_1),\ldots, (x_n,y_n)$ (where $x_i\neq x_j$ for $i\neq j$) we can use the least squares method to obtain a polynomial $f$ of degree $m$ (with $m<n$) such that the ...
- 21
0
votes
0
answers
57
views
Gradient of $w \mapsto\|y-Xw\|_2^2$
When I try to find the gradient of the MSE loss $$L(w)=\|y-Xw\|_2^2$$ (ignoring constant factors) I find two different solutions, one is the transpose of the other:
Compute the gradient of mean ...
- 1
3
votes
1
answer
37
views
Linear system. I know coefficients and solutions, I want to approximate inputs.
This may be a simple question, but I couldn't find a solution because I didn't know what to search for. Excuse my math.
I have a linear system $AX = Y$. $A$ is an affine transform and all the ...
- 33
1
vote
0
answers
14
views
Solution of RBF constraint least squares
I have to design a matrix A that solves a linear system:
$𝑦=𝐴𝑥$, where x is a known complex vector and y is a known complex vector.
The requirement is that A is an RBF kernel, i.e., it has the ...
- 59
6
votes
2
answers
66
views
Given a matrix $A$ and vector $x$, is it possible to find the permutation of $x$ that minimizes $\lVert APx \rVert$?
Let $A$ be some $n\times n$ matrix, and $x$ be a column vector of length $n$. I am looking to find the permutation matrix $P$ of size $n\times n$ that minimizes the vector norm of $APx$.
There are $n!$...
- 71
2
votes
0
answers
51
views
$[1]$ Does good $R^2$ necessarily tells us that the experiment went right? $[2]$ Is not extrapolation dangerous when used for a far point?
In an experiment, the relation between $x$ and $y$ is linear.
$x_\text{actual} = \alpha_x x_\text{observed} + \beta_x + \gamma_x$. [Here, $\alpha$ is the slope, $\beta$ is the intercept, and $\gamma$ ...
- 4,800
0
votes
0
answers
22
views
Solutions of linear equations with singular parameter matrices
I've been trying to solve a least square problem with a singular parameter matrix: solve x from $Ax=b$ with $A$ being singular. It has been confirmed that infinitely many solutions of $x$ in $Ax=b$ ...
0
votes
1
answer
39
views
Least square representation using eigenvector of $E[xx^T]$
The question in the context of random design. E represents expectation.
Define $\beta$ such that $E[(\langle\beta, x\rangle - y)^2] = \min E[(\langle w, x\rangle - y)^2]$
Then we can get $\beta:= \sum ...
- 337
0
votes
0
answers
27
views
Equivalence between first-order optimality conditions for rank-$1$ constraint and quadratic form
When simplified down, my optimization problem has the following structure
$$\underset{x\in\mathbb{R}^n}{\arg\min} \quad \left\| b - A\operatorname{vec} \left( x x^\top \right) \right\|_2^2$$
I am ...
- 71
0
votes
0
answers
23
views
A parametric model for inverse of quadratic equation
I have data $\left\{ q_i, p_i \right\}_{i = 1}^{N}$ from a model $p_i = a q_i^2 + b q_i + c$:
I know data is always in the positive quadrant as $q_i \geq 0, \, p_i \geq 0$ and I also know $a > 0$ (...
- 109
5
votes
1
answer
256
views
Sparse Cholesky decomposition of factorized matrix
I want the diagonal of a matrix $Y^TA^{-1}Y$ where $A=X^TX$ and $X$ is very sparse with dimensions ~1e6 x ~1e5 (so $A$ is 1e5 by 1e5). $Y$ is something like 1e5 by 1e4 (also sparse). Currently I'm ...
- 153
0
votes
0
answers
11
views
Non-negative Least Square over the Unit Simplex. Complexity analysis.
I want to solve the non-negative least square problem with equality constraint, e.g.
$\min ||Ax-b||_2^2$ s.t. $x\geq 0, 1^Tx=1$.
There are some great posts on stackexchange, that elaborate how ...
- 695
10
votes
3
answers
340
views
Determining the distance to the origin from three points on a circle
Scenario:
A measuring tool consists of a round measuring bung (red circle) equipped with three probes (labeled $a$, $b$, and $c$). These probes are evenly spaced at angles of $120^\circ$ from each ...
- 203
0
votes
0
answers
57
views
Method of Least square
Sometimes, the method of least squares can be used for problems that, in principle, could not be addressed by the method.
For example, let's suppose we want to find $a$ and $b$ in order to fit the ...
- 29