Skip to main content

Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

Filter by
Sorted by
Tagged with
1 vote
0 answers
34 views

Finding the least squares solution of a linear system based on a QR factorization

One method of finding the least squares solution of the following "augmented system" $$ \left[ \begin{matrix} I & A \\ A^T & O \end{matrix} \right] \left[ \begin{matrix} r \\ x \end{...
Olumide's user avatar
  • 1,261
0 votes
1 answer
51 views

Bounds on error on solving $Ax=b$ for perturbed system

I am interested in solving $x$ from $Ax=b$, where $A \in \mathbb{R}^{m \times n}$ and $m > n$, it is rectangular and even rank-deficient. This means that we can only solve it in the least-square ...
William Lin's user avatar
0 votes
0 answers
44 views

Least Squares Function Approximation and Convexity of Functions

I have been reading about Least Squares function approximation and am dealing with the following definition: Let $f$ be continuous on $[a,b]$ and let $W$ be a finite dimensional subspace of $C[a,b]$. ...
MattKuehr's user avatar
  • 197
0 votes
0 answers
46 views

Least squares polynomial of degree 2

Question: Population growth in imaginary districts is described by the following table. Use the least-square polynomial of degree $2$ to predict populations for the year $2000$. $$\begin{array}{c|...
Kai's user avatar
  • 1
1 vote
0 answers
22 views

Understanding error estimate for least squares fit.

I'm reading the following paper link. In it, on page $7011$, the authors apply linear least squares to a system in the form $$ {\bf A}x = b$$ Where $\bf A$ has $M$ rows and $N$ columns, they state an ...
foobar's user avatar
  • 11
1 vote
0 answers
22 views

Proving OLS estimator from two linear models

In a paper I'm reading, the authors try to recover an uncontaminated signal $x$ from two time series of imaging data, $y_1$ and $y_2$, which follow the relationships below: $y_1$ = $x$ + $z$ + $\eta$ $...
malortncachaca's user avatar
1 vote
0 answers
57 views

An alternative to Levenberg–Marquardt algorithm

When trying to solve for a (over)determined non-linear least square method: $$\underset{x}{\min}||f(x)||^2_2, f: \mathbb{R}^n \rightarrow \mathbb{R}^m, x\in \mathbb{R}^n, m\geq n$$ we use the Gauss-...
William Lin's user avatar
5 votes
1 answer
58 views

Given $A_i, B_i \in \mathbb{R}^{k \times d}$, minimize $\sum_{i} \lVert U A_i V^T - B_i \rVert_F^2 $ over orthogonal $U, V$.

Given a collection of rectangular matrices $A_i, B_i \in \mathbb{R}^{k \times d}$ for $1 \leq i \leq n$, I am looking for an analytical solution for orthogonal matrices $U \in \mathbb{R}^{k \times k}$ ...
tommym's user avatar
  • 421
1 vote
1 answer
86 views

Prove that PCA decomposition captures all information in a factor model

Assume a data matrix $X \in \mathbb{R}^{N \times p_X}$. Let it have some exact lower dimensional factor representation $X = A F$, where $F \in \mathbb{R}^{N \times p_F}$ and $p_F < p_X$. Let the ...
fes's user avatar
  • 1,671
0 votes
0 answers
53 views

How does Self-Scaling Fast Givens QR work for (Regularized) Linear Least Squares

Basic Problem I need help to understand the Self-Scaling Fast Givens QR decomposition proposed in Anda A. A. & Park H., Self-Scaling Fast Rotations for Stiff and Equality Constrained Linear Least ...
MothNik's user avatar
1 vote
1 answer
44 views

Bound on norm of least squares solution

Suppose we have a linear system $Ax = b$ where $A$ is $n \times m$ for $n > m$ and has full column rank. The least squares solution is given by $x = (A^T A)^{-1} A^T b$. What upper bound can be ...
wrb's user avatar
  • 11
1 vote
1 answer
28 views

Given an inconsistent overdeterminate system AX=b where $A\in M_{m×n}(R)$ and $b\in R^m$ with rank A=n. Find the least square approx. solution of AX=b

Suppose $A$ is a real matrix of order $m\times n$ with $m>n,b\in\Bbb R^m$ be such that the over determined system of linear equations $AX=b$ is inconsistent and $\text{rank} (A)= n.$ Let $W$ be the ...
Thomas Finley's user avatar
1 vote
0 answers
32 views

Verifying Math behind Full Bundle Adjustment for Multi-Camera Extrinsic Calibration

I'm working on an extrinsic calibration problem involving multiple monocular cameras. Each camera has known intrinsic parameters, and I have captured timestamp-aligned images of a calibration target ...
hunterlineage's user avatar
1 vote
2 answers
103 views

Is there a method to fit a vertical hyperbola to a set of points?

I'm trying to fit a vertical hyperbola to scattered points. I'm trying to follow this article "Direct and specific least-square fitting of hyperbolæ and ellipses” This question is in the context ...
user1325242's user avatar
2 votes
1 answer
127 views

Find least squares system of linear equations

Find a real system of linear equations $Ax = b$ where A is 2 columns and 4 rows matrix and all elements of A are not zero, $b \in \mathbb{R}^4$, $z = (2\ 3)^T$ is the approximation of $Ax = b$ using ...
meerkat's user avatar
  • 358
3 votes
1 answer
55 views

Treat the sum of residuals as scalar function for Gauss-Newton least squares regression

I have recently learned about Gauss-Newton method and puzzled about something. So given the model function $f(\vec{x}, \vec{\beta})$, and data points $(\vec{x_i}, y_i)$, the residuals are $S = \sum_{i}...
baronett's user avatar
0 votes
0 answers
11 views

How to optimize the number of data points used to interpolate a fit over an unknown polynomial (with noisy data)?

I'm trying to find the coefficients of an unknown polynomial given that I can choose to extract a coordinate at any point but at a cost. The cost goes up as more points are chosen. When a coordinate ...
Zinn's user avatar
  • 101
0 votes
0 answers
12 views

Find the expected value of the sum of squares of the residuals.

Question: In $\mathbb{R}^2$, it is given $n$ points $(x_i,Y_i)$, where $x_i$ are known constants, and given independent random variables $Y_i\sim U(l_i,r_i)$. Fit a straight line through these $n$ ...
Hexarhy's user avatar
  • 11
1 vote
0 answers
23 views

Does this proof that $\min_P ||APx||$ is NP-complete (for general $n\times n$ $A$, permutations $P$, and a vector $x$) work?

Let $A$ be some $n×n$ matrix, and $x$ be a column vector of length $n$. I am looking to find the permutation matrix $P$ of size $n×n$ that minimizes the vector norm of $APx$. In a previous question ...
T. Zaborniak's user avatar
1 vote
0 answers
14 views

A corollary of Frisch-Waugh-Lowell Theorem

The formulation is just a special case of FWL. Say we have a partitioned regression, $Y=X_1\beta_1+X_2\beta_2+\epsilon$ but with $X_2$ be $n\times 1$ and $\beta_2$ a constant. Let $b_1,b_2$ be two OLS ...
Chang Henry's user avatar
0 votes
0 answers
46 views

Estimating parameters in a system of ODE's from data.

I'm trying to model the COVID-19 pandemic in Madrid during the first wave, to model the data I'm using the SIR model. The SIR model stands for suceptible, infected and recovered. It is a compartmental ...
Matin Gomez-Pablos's user avatar
1 vote
1 answer
44 views

Orthogonal procrustes with kernel constraint

Given a matrix $M \in \mathbb{R}^{n \times m}$ with $n > m$ and an arbitrary vector $v \in \mathbb{R}^n$, I am looking for an analytical solution for the orthogonal matrix $R \in \mathbb{R}^{n \...
tommym's user avatar
  • 421
3 votes
1 answer
71 views

Least square derivatives

Let $X_1, \ldots, X_N \in \mathbb{R}^p$ and $Y_1, \ldots, Y_N \in \mathbb{R}$. Define $$ X=\left[\begin{array}{c} X_1^{\top} \\ \vdots \\ X_N^{\top} \end{array}\right] \in \mathbb{R}^{N \times p}, \...
user avatar
3 votes
1 answer
111 views

Solve $\| X A - B \|$ subject to $X C = C X$

Given $A, B \in \mathbb{R}^{n \times k}$ and S.P.D. $C \in \mathbb{R}^{n \times n}$, I would like to find an analytical solution for the matrix $X \in \mathbb{R}^{n \times n}$ that minimizes \begin{...
tommym's user avatar
  • 421
0 votes
1 answer
166 views

Can a (bounded) linear least-squares problem include a scale factor in its solution?

I have a system of equations. $$ \small \begin{aligned} (1 - p_0)u_0 + (q_0 - 1)v_0 + u_1 - v_1 = r_0 - c t_0 \\\\ (1 - p_1)u_1 + (q_1 - 1)v_1 + u_0 - v_0 = r_1 - c t_1 \\\\ (1 - p_2)u_2 + (q_2 - 1)...
Isco's user avatar
  • 103
0 votes
1 answer
57 views

Symmetric linear least-squares solution with known diagonal elements

Given matrices $\pmb{A}\in\mathbb{R}^{p\times n}$ and $\pmb{B}\in\mathbb{R}^{p\times n}$ with $p>n$, I need to solve the following linear system in symmetric matrix $\pmb{X}\in\mathbb{R}^{p\times p}...
Hepdrey's user avatar
  • 83
1 vote
1 answer
90 views

Minimizing Frobenius norm involving inverse

I am looking for methods to solve the following minimization. Let $A\in \mathbb{R}^{n\times k}$, $B\in \mathbb{R}^{n\times m}$, $C\in \mathbb{R}^{m\times m}$ and $E\in \mathbb{R}^{m\times k}$, where $...
dff's user avatar
  • 113
0 votes
1 answer
44 views

Taking partial derivative of SSD, wrt the parameters $a, b$?

In the book titled: Analysis of Straight-line data, by Forman S. Acton; it is given on page #10: The classical 'least squares' procedure is most commonly derived by forming an expression for the sum ...
jiten's user avatar
  • 4,532
0 votes
0 answers
10 views

Variance at a single measurement

I have a collection of data points, $(x_0, y_0)...(x_{n-1}, y_{n-1})$ of the function $y = y(x)$ where the values of x are in ascending order. I'm working out an algorithm for smoothing splines, ...
OrangeWombat's user avatar
5 votes
2 answers
333 views

Maximization of Linear Least Squares with a Triangular Matrix over The ${L}_{2}$ Unit Ball

I have the following optimization \begin{align} \max_{\|x\|^2\le1} \|Lx - y\| \end{align} where $L$ is a lower triangular and $y$ is a given vector. Does it admit a closed-form solution? I am ...
Morad's user avatar
  • 547
1 vote
1 answer
34 views

SVD and least square solution

Let $K \in \mathbb{R}^{m,n}$, $u \in \mathbb{R}^n$, and $f \in \mathbb{R}^m$. Assume that $m < n$ and $K$ have full rank so a solution exists but is not unique. I want to understand why this ...
endeavor's user avatar
  • 311
0 votes
0 answers
21 views

Convergence of coefficients in multivariate regression

In this thread, the convergence of coefficient for univariate dependent variable is proven. I wonder, assuming the same setup, how can the convergence be extended to multivariate as: $$Y=XW+\epsilon$$ ...
statwoman's user avatar
  • 253
0 votes
1 answer
50 views

Linearly spacing points along non uniform curve

For a robotics project, we have sensors that give us an accurate realworld XYZ position. I've recorded this data to measure the sag our telescoping boom has over its 50 feet of travel. This is the raw ...
Joe Jankowiak's user avatar
0 votes
0 answers
19 views

Property of projection matrix $P = X(X'X)^{-1}X'$ under random design context

In OLS with fixed design, we project $Y$ onto the column space of design matrix $X$, which is $\mathcal{C}(X)$. The residual $Y-PY = (I-P)Y$ is orthogonal $\mathcal{C}(X)$. Further more, $(I-P)$ is a ...
maskeran's user avatar
  • 573
-1 votes
1 answer
68 views

Binary solution to least squares linear regression [closed]

I am looking for a closed form solution $x^*$, binary vector, to $$\arg\min_{x}(\|M x + b\|_2),$$ restricted to $x \in \{ 0,1 \}^n$. Here $b \in \mathbb{R}^{m}, M \in \mathbb{R}^{m \times n}$ are ...
Yeb02's user avatar
  • 11
1 vote
1 answer
33 views

How to find a matrix perturbation which lowers the rank of a matrix

I have a matrix $A \in \mathbb{R}^{m x n}$ which has independent columns. I want to find the smallest perturbation which will make it have a kernel and a vector in that kernel. Something like $$ \min_{...
Mark's user avatar
  • 121
0 votes
0 answers
22 views

Solving for a conjugation matrix from observed transformations

I am trying to find the matrix $M$ satisfying: $$v_t'= MA_tM^{-1}v_t$$ For a dataset of observed transformations $(v_t',A_t,v_t)_t$. Basically I have two isomorphic vector spaces $U$ and $V$, where ...
hamza keurti's user avatar
0 votes
1 answer
25 views

Find a matrix that maps a several broad region in $R^n$ to a small regions in $C^n$

I have several vectors $y_1, y_2, y_3 \cdots y_n \in R^n$ that I need to linearly map to a single vector $x_0 \in C^n$. The same matrix should also map $z_1, z_2, z_3, \cdots z_n \in R^n$ to a single ...
user3284182's user avatar
4 votes
0 answers
81 views

Numerical Least squares estimation on the norm of the minimiser

I was looking at Proposition 3.2. in Applied Numerical Linear Algebra by JW Demmel, that states that when solving $$\min \|Ax - b\|_2 $$ if in the singular value decomposition $\sigma_{min}>0$ then ...
L. Schiavone's user avatar
1 vote
1 answer
115 views

How to partially differentiate an equation with sums?

I'm working on a simple linear regression model in a physics course, where we are doing measurements of the round trip speed of light, over increasing distances. We are using the Least Squares method ...
THH's user avatar
  • 13
1 vote
2 answers
45 views

An iterative method for minimizing squared errors between nonlinear functions

Suppose that $f$ and $g$ are two vector-valued functions, where $f$ has a possibly highly complex nonlinear form and $g$ is much simpler (e.g., can even be linear). Consider the nonlinear least ...
Miles N.'s user avatar
  • 157
1 vote
0 answers
18 views

Linear least squares involving linear functionals

Let $\Omega\subset\mathbb{R}^d$ be a domain. Suppose $H$ is the second order Frechét derivatives of a function $f\in L^2(\Omega)$, then $H\in (L^2\times L^2)^{*}$. Let $g\in (L^2(\Omega))^{*}$ and we ...
Nicolas's user avatar
  • 179
0 votes
1 answer
33 views

Question about finding slope from my textbook, walking through linear regression through the least squares method (pre-calculus)

I have question about a specific step describing how to do linear regression with the least squares method. In the textbook, it says that you need to find $\triangle x$ and $\triangle y$ (which are $...
B Karr's user avatar
  • 1
0 votes
1 answer
87 views

Deriving the least squares method

I would like to understand more about the Least Squares, but it remains unclear to me, where these equations come from. for a linear approximation $y=ax+b$ $a=\frac{m\sum_{i=1}^{m} x_iy_i-(\sum_{i=1}^{...
einzigartigerhummer's user avatar
3 votes
1 answer
77 views

Gauss-Newton method, where did sigma inverse come from?

I'm studying the Gauss-Newton Method from "slambook-en" chapter 5 on optimization (the books is made free online by the author in case you need to see it). I've attached a picture of the ...
JerSci's user avatar
  • 59
0 votes
1 answer
28 views

Non-negative autoregressive model

I am trying to fit a linear 1-order autoregressive model to some multivariate time-series data. The model I am using is of the form $$x_t = Ax_{t-1}+\xi_{t-1}$$ and I am solving it in R using the mAr ...
citizenfour's user avatar
2 votes
0 answers
64 views

I'm a bit confused on how to linearize an Exponential Graph

I've seen other people answer questions on how to do this, I tried it and it didn't work. As of right now, I'm using the following desmos graphs. Desmos, Exponential Graph Desmos, Linearized ...
Yashwak's user avatar
  • 15
0 votes
1 answer
62 views

Fitting trigonometric function

Let $y$ be a measured time series, and $\hat{y}(t) = A\cos(\omega t +\phi)$. How to find $A$, $\omega$, $\phi$ that minimizes $$\sum_{k = 0}^{N - 1}|\hat{y}(k) - y_k|^2$$ The amplitude spectrum looks ...
user877329's user avatar
0 votes
0 answers
32 views

Solve linear IV-GMM numerically

I'm interesting in solving a linear IV-GMM (see page 5 and 6 for background). The solution takes the form $$ \hat{\beta} = (X'ZWZ'X)^{-1} X'ZWZ'y $$ where $W$ is a positive definite weighting matrix ...
Giacomo's user avatar
  • 147
1 vote
1 answer
90 views

Closed form solution for least squares with linear constraints problem

I'm trying to figure out if the following linear program has a closed form solution $$ \begin{align} \min_{x \in \mathbb{R}^{n}} ||x&-\alpha||^2 \\ Ax &= b \\ x &\geq 0 \end{align} $$ ...
Dalamar's user avatar
  • 757

1
2 3 4 5
38