Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

Filter by
Sorted by
Tagged with
0
votes
1answer
25 views

Fitting a line through intercept 0

I need to code a least squares routine to fit a line $$y = m*x$$ into a 2d set of points $$(x_i,y_i)$$ How can I find the regression line without an interceptor?
1
vote
0answers
17 views

Least Squares Intersection between multiple line segments

I'm wondering how I would go about computing the 'best fitting' intersection between multiple line segments (or even better lines of bearing) using the least squares method. I understand how to use ...
0
votes
1answer
41 views

Understanding the Gauss-Newton Method

I have successfully implemented the Gauss-Newton method to a simple nonlinear least-squares problem as shown in the Wikepedia page here. As I understand it, the method uses the derivative of the ...
0
votes
0answers
21 views

Convexity of linear least squares if derivative of coefficients is added to objective

My math background is quasi non-existent, so please bear with me. Context: I am implementing a method for spectral unmixing called MCR-ALS (Multivariate Curve Resolution - Alternating Least Squares) ...
0
votes
3answers
61 views

Showing Normal Equations as Linear System of Equations

[This is a practice problem] I watched tutorials on least square method and normal equations and understood them too. However, i am confused with this question: Measurement vals $p_0 = 0, p_1 = 2$,...
0
votes
0answers
21 views

Linefitting to 6 dimensional points

I would like to find a line best-fitting an arbitrary set of input points in 6 dimensions. Is there an efficient algorithm to accomplish this. Does the usual linear least square approach work here?
0
votes
0answers
9 views

How to do regularization onto a vector?

Assume that we have a vector $x(k)$ that contains noise. We don't know the noise. Now we want to do regularization onto $x(k)$ so it will become more...clear. Is that possible? I assuming that it ...
0
votes
0answers
16 views

What method should I use if I want to find best fit for a matrix $B$ inside matrix $A$

Assume that we have a real matrix $A$ with the size $n,m$. Then we have a real matrix $B$ with the size $i,j$ where $i << n$ and $j << m$. In other words, $B$ is much smaller than $A$. ...
0
votes
0answers
26 views

Hessian of least squares problem

Consider the least squares problem $f(x)=1/2*\Sigma_{i=1}^{m}{e_i(x)^2}$. For the gradient of $f(x)$, I could understand that it equals to: But I do not understand the calculation of Hessian: Where ...
0
votes
1answer
34 views

Given an $n\times m$ matrix $A$ prove that $Col(A^TA)=Col(A^T)$

We're studying Least-Squares in my linear algebra class; one of the theorems we've proven is that $Nul(A^TA)=Nul(A)$. I understood that proof well-enough, however our professor also stated that $Col(...
1
vote
1answer
45 views

Solving a Variation of Linear Least Squares - $\arg\min_{x} \frac{1}{2} {\left\| \left( \sum_{i} {A}_{i} x {b}_{i}^{T} \right) - C \right\|}_{F}^{2}$

How to solve the following variant of Linear Lieast Squares problem: $$ \arg \min_{x \in \mathbb{R}^{n}} \frac{1}{2} {\left\| \left( \sum_{i} {A}_{i} x {b}_{i}^{T} \right) - C \right\|}_{F}^{2} $$ ...
2
votes
1answer
55 views

Linear Least Squares Problem of a Specific Matrix Form

Given $A_i\in\mathbb{R}^{k\times N},i=0,1,...,M,$ with $k>N$, and $b_i\in\mathbb{R}^k, i=1,2,...,M$. Consider the following least square problem $$ \begin{bmatrix} A_0 & 0 & \cdots &...
1
vote
1answer
38 views

Least square using orthogonal polynomial

I have obtained some orthogonal polynomials, using Gram orthogonal process, and the next question says, using them (O.P.) obtain the least square approximation of second degree for $f(x)=x^{3/2}$ on $[...
0
votes
0answers
21 views

Least-Squares System Identification of combined waveshaper + FIR system

I have a system that consists of a polynomial waveshaper $s(x) = s_0 + s_1x + s_2x^2 + \dots + s_lx^l$ followed by a causal FIR filter with coefficients $h = \begin{bmatrix} h_0 & h_{1} & h_{...
0
votes
1answer
32 views

Difference Between Matlab / Octave Solve and Numpy Solve

I am trying to solve a linear system on Python for a problem which I ported from Octave. I have the following code for Octave ...
0
votes
1answer
23 views

Approximate a and b of $y = ae^{bx}$ using the method of least squares

My data are: x 2.2 2.6 3.4 4.0 y 65 61 54 50 If I take the logarithm of the given equation, I will get $$\ln y = \ln a + bx$$ How should I use my data from here?...
0
votes
0answers
21 views

Attempting to project 4 values, to 5 values points to calculate an approximate average of 4 point values in 5 point value space.

Attempting to project 4 values, to 5 values points to calculate an approximate average of 4 point values in 5 point value space. The problem: Attempting to map a 4 point proficiency grading to a 5 ...
0
votes
0answers
43 views

Gradient Descent convergence - multivariate regression

Follow up to this post: Does gradient descent converge to a minimum-norm solution in least-squares problems?, but with all matrices Suppose we are given $p \times n$ matrix $\mathbf{X}$ and $q \times ...
1
vote
0answers
19 views

Transforming a regression model with dummy into two models

Consider the following regression model: \begin{equation} y = \alpha + \beta X + \gamma D + \epsilon, \end{equation} where $y$ is the dependent variable, $X$ is a continuous variable, $D$ is a dummy ...
-2
votes
1answer
129 views

How to Solve Linear Least Squares Problem with Box Constraints [closed]

How could one solve the following Least Squares with Box Constraints problem: $$\begin{aligned} \arg \min_{x \in \mathbb{R}^{n}} \quad & \frac{1}{2} {\left\| A x - b \right\|}_{2}^{2} \\ \text{...
1
vote
1answer
25 views

least square estimators and simple linear regression related proofs

Suppose that we have independent samples ${(x_i,y_i ):i=1,⋯,n}$ which are assumed to follow $ y_i=β_0+β_1 x_i+ε_i $ where $\epsilon_i$ are i.i.d. from $N(0,\sigma^2)$ . Suppose that $b_0$ and $b_1$ ...
0
votes
1answer
17 views

proofs based on least square estimators (simple linear regression)

Suppose that we have independent samples ${(x_i,y_i ):i=1,⋯,n}$ which are assumed to follow $ y_i=β_0+β_1 x_i+ε_i $ where $\epsilon_i$ are i.i.d. from $N(0,\sigma^2)$ . Suppose that $b_0$ and $b_1$ ...
1
vote
1answer
41 views

Can someone explain to me least squares regression using vectors?

I am reading Foundations and Applications of Stats by Pruim. I am having a tough time understanding least squares regression from a vector notation standpoint. I don't have a good understanding of ...
0
votes
0answers
21 views

Solution of matrix equation with only partly known matrices

Is it possible to solve the following matrix equations for $X$: $ H_1=A_1 \ast X \ast B_1 \\ H_2=A_2 \ast X \ast B_2$ with $\left \{ A,B,H,X \right \}\in \mathbb{C}^{4\times 4}$ $A$ and $B$ are ...
1
vote
1answer
32 views

fitting a 1d function with an asymptote

I am trying to fit a positive function $f(x)$ , $0 < x\leq 1$, with the following properties: $$f(0) =\infty$$ $$f(1) = 1$$ $$\frac{df}{dx}(1) > 0$$ $$f(x) > 0,\text{ }\forall\text{ ...
-1
votes
1answer
32 views

Reformulation of matrix optimization. Is it equivalent?

Consider the linear least squares problem: $${\bf x_o}=\min_{\bf x}\|{\bf Mx-b}\|_2^2$$ can be solved by normal equations: $${\bf x_o} = ({\bf M}^T{\bf M})^{-1}{\bf M}^T{\bf b}$$ Assuming $\bf M$ ...
1
vote
0answers
23 views

Least Squares removing first $k$ observations Woodbury formula

Given the matrix $X_{n,p}$ from the least squares problem $$ \mathbf{X} \cdot \mathbf{\beta} = z $$ Where the normal equation is: $$ \mathbf{\hat{\beta}} = \left(\mathbf{X}^T \mathbf{X}\right)^{-1} ...
0
votes
0answers
19 views

Error term covariance of two-stage estimators

I am looking into properties of two-stage estimators (2SLS). My setting is as follows: 1) $y_1 = y_2\beta+\epsilon$ 2) $y_2 = z \pi_2 + \eta$. Equation 1 represents the second-stage estimation, ...
0
votes
3answers
45 views

In least squares estimation, why are the residuals constrained to lie within the space defined by the following equations?

I've been reading through the Wikipedia article on degrees of freedom (statistics). There is a section about residuals, in relation to least squares estimation. The article says: Suppose you have ...
0
votes
0answers
32 views

Variance of Least Squares Estimate

Universally the literature seems to make a jump in the proof of variance of the least squares estimator and I'm hoping you can fill in the gaps for me. In matrix form, the least squares estimate is: ...
2
votes
2answers
80 views

Least squares solution for 2 perpendicular lines in vector notation

I have the convex hull of the corner of a building in 2D and I am trying to fit 2 perpendicular lines to the set of points on that hull to get me the orientation of the corner. I have a solution but ...
0
votes
0answers
69 views

Projection of $z$ onto the affine halfspace $\{x \mid Ax=b, \; x>0\}$.

This is exactly the same question as this one exept i want to project on only a half affine space instead of a full affine space. The parameters $A$ and $b$ of the problem are as in the linked ...
2
votes
0answers
21 views

Necessary and sufficient condition for two components of least square estimate to be negatively correlated

Consider a linear model in a matrix form $Y = X\beta + \epsilon$ where $Y$ is a response vector, $X$ is a $n$ by $p$ ($p < n$) full rank matrix of predictors, $\beta$ is a parameter vector, and $\...
0
votes
0answers
22 views

Is β_0 unbiased and consistent?

I have the following equation $y_i=β_0+β_1 x_{1i}+β_2 x_{2i}+e_i √(exp⁡(x_{2i}))$ I found that theoretically $β_0$ should be unbiased because: Would this explanation be correct? That said I'm ...
3
votes
1answer
157 views

Least Squares and perturbations

Consider $Y = \theta X + \Delta$ where $\theta$ is the unknown matrix to be found, $Y, X$ are the data matrices of finite length $T$ $Y = [y_0 \ y_1 \ \dots \ y_T], \ X = [x_0 \ x_1 \ \dots \ x_T]...
3
votes
2answers
265 views

Questions about norm of $\mathbf{x}$ when minimizing squared norm of $\mathbf{A} \mathbf{x} - \mathbf{b}$

Section 4.5 Example: Linear Least Squares of the textbook Deep Learning by Goodfellow, Bengio, and Courville, says the following: Suppose we want to find the value of $\mathbf{x}$ that minimizes $$f(\...
0
votes
1answer
32 views

Polynomic curve interpolation with tensor differential constraints?

Inspired by this question and these lecture notes which belong to it, where derivatives of specific points is prescribed to be some known vectors. Let us consider the $${\bf c}'(0) = {\bf v_0}\\{\bf ...
3
votes
3answers
342 views

Minimizing linear least squares using Lagrangian $L(\mathbf{x}, \lambda) = f(\mathbf{x}) + \lambda (\mathbf{x}^T \mathbf{x} - 1)$

Section 4.5 Example: Linear Least Squares of the textbook Deep Learning by Goodfellow, Bengio, and Courville, says the following: Suppose we want to find the value of $\mathbf{x}$ that minimizes $$f(\...
1
vote
0answers
22 views

Inverse to a block-matrix, how to utilize knowledge together with conjugate-gradient?

Consider the following matrix: $${\bf M} = \begin{bmatrix}a&b\\0&c\end{bmatrix}$$ if $a,b,c \in \mathbb R$, we can find: $${\bf M}^{-1} = \begin{bmatrix}1/a&-b/(ac)\\0&1/c\end{...
1
vote
2answers
83 views

Approximate ar function so that it becomes linear in parameters (without Taylor)

TL;DR (original question): I am looking for a function that has roughly the form of $$f(x) = \exp\left(−(x/a)^2\right) − \log\left((x/a)^2+1\right)$$ but is linear in its model parameter (here $a$) ...
2
votes
1answer
75 views

Least Squares using QR for underdetermined system

I am trying to understand the usage of $QR$ decomposition for Least Squares problem of $$ Ax=b$$ when the system is underdetermined - $A$ is $m\times n$ and m < n, but $A$ is full rank and the ...
0
votes
0answers
33 views

Are vectors in the range space unique?

Consider u to be the solution to a least squares problem, $$\arg\min \frac12 \| Ax - b \|^2$$ u = v + w , where v belongs to Range(A) and w belongs to null(A) If I want to prove that the solution ...
0
votes
1answer
41 views

Independence of Least Squares Estimators (LSE)

Consider the linear regression equation $y_i = \beta_0 + \beta_1x_i + \beta_2x_i^2 + \epsilon_i$ in the quadratic polynomial model with the independent normal noise $\epsilon_i \sim N(0, \sigma^2)$. ...
1
vote
2answers
82 views

Regularized Least Squares Objective - Sufficient and Necessary Conditions for Unique Solution

Consider some form of Tikhonov regularization, where we seek to minimize the objective described by: $\min_{x \in \mathbb{R}^n} ||Ax - b||^2_2 + \lambda||Dx||^2_2$ (with matrices $A, D$ and vectors $...
0
votes
1answer
28 views

Matrix algebra properties square of $Ax-b$

I have seen the least squares formula derived like this : $f(x) = ||Ax-b||_2^2 = \\ (Ax-b)^2 =\\ x^TA^TAx-2b^TAx+b^Tb\\ \nabla f(x) = 2A^TAx -2Ab = 0 => x=(A^TA)^{-1}A^Tb$ I'm trying to derive ...
2
votes
1answer
60 views

Finite Derivative of $\frac{x^3-1}{x+1}$

I have the following qeustion: Evaluating the front, back and central derivative I just need to plug it into the corresponding: $$D_{+}(x)=\frac{f(x+h)-f(x)}{h},D_{-}(x)=\frac{f(x)-f(x-h)}{h},D_{c}...
0
votes
1answer
65 views

Show that $\| \mathbf{b} - \mathbf{A} \mathbf{x} \|_{2}^{2} < \| \mathbf{b}\|_{2}^{2}$ given $|x_{i}| = 1 \quad \forall i$

Suppose that I solve the following optimization problem: $$ \underset{\mathbf{x}}{\text{minimize}} \quad || \mathbf{b} - \mathbf{A} \mathbf{x} ||_{2}^{2} $$ $$ \text{subject to} \; \; |x_{i}| = 1 \...
1
vote
0answers
17 views

Determine missing points of plotted curve.

I have a scientific study with charts as the one in the image. They made a curve to merge data from different studies and methods... using the equations in picture. I have access to the resulted ...
1
vote
0answers
39 views

Least-squares constrained to the unit Euclidean sphere [duplicate]

Assume $A\in \mathbb{R}^{t\times n}$ and $b\in \mathbb{R}^t$. How to solve the following optimization problem in $x\in \mathbb{R}^n$? $$\begin{array}{ll} \text{minimize} & \|Ax-b\|_2^2\\ \text{...
0
votes
1answer
41 views

Approximating a sigmoid curve from noisy data

Given a set of points $\{(x_0,y_0),(x_1,y_1),...,(x_n,y_n)\}$ where $0\leq x_i < 1$ and where the $y_i$ are noisy what method can be used to find a smooth, monotonic, sigmoid-like function to ...

1 2
3
4 5
28