Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

1
vote
1answer
22 views

Using the least squares method for problems with two independent variables

This may be quite a specific question and I apologise however I have struggled to find any information regarding a method. I have 6 given $P_i$ values and 6 given $E_i$ and $F_i$ values and I want to ...
0
votes
0answers
39 views

In this constrained minimization problem, should the Lagrange multipliers be positive?

Consider the following (real, block ?) matrix $Z_{n\times k+1}=[1_{n\times 1},X_{n\times k}]$. Note how $z\equiv v^TZZ^Tv$ can be written as: $v^T11^Tv+v^TXX^Tv=v^TJ_nv+v^TWv$, where $J_n$ is a unit ...
0
votes
0answers
25 views

OLS estimator of AR($1$) is biased

Suppose that we have a sample $X_0,X_1,\ldots,X_n$ from the AR($1$) model given by $$ X_t=\phi X_{t-1}+\varepsilon_t $$ for $t\in\mathbb Z$, where $|\phi|<1$ and $\{\varepsilon_t\}_{t\in\mathbb Z}$ ...
0
votes
1answer
10 views

Househbolder transformation identity matrix dimensions

When performing a householder transformation and generating an elementary reflector matrix of the form: $$H = I - 2\dfrac{vv^T}{v^Tv}$$ How do we know the dimensions of the identity matrix?
-1
votes
1answer
23 views

Why can we replace dependent variable y with the residuals/error term e?

I don't understand why we can replace $y$ with $e$: As in the proof of the Gauss-Markov theorem, $$ \tilde{\beta} = [\,(W' W)^{-1}W' + C\,][\, W\beta + e\,] = \beta + (W' W)^{-1}W'e + CW\beta + ...
0
votes
0answers
11 views

Unclear proof in the Gauss-Markov theorem

I don't understand a step in the proof of the Gauss-Markov theorem: Mainly, why can we simply replace y with e, given that y is defined as: Thanks in advance!
0
votes
0answers
17 views

Which method is better approximation in a given case?

Let $Q(x)\in\mathbb{R}[x]$ be a polynom. Let 4 sample points out of $Q(x)$: $$ x_0=-1,x_1=0,x_2=2,x_3=3, \\Q(x_0)=0, Q(x_1)=-5, Q(x_2)=15,Q(x_3)=64$$. a) Based on the given points, build an ...
0
votes
1answer
13 views

How to find the best quadratic approximation to a cubic function between two of its (real) roots

Consider a real cubic polynomial function with three real roots. By applying a translation and horizontal and vertical dilations, we can transform it to the form $$p(x) = x (x - 1) (x - r)$$ for ...
0
votes
0answers
16 views

Inverse of matrix sum, one symmetric PSD and one near-constant diagonal

Question How can split the calculation of a real matrix inverse $(S + D)^{-1}$ when I know that $S$ is symmetric and PSD and $D$ diagonal with only a handful of unique values (=diag$(a,a...a,b...b,c.....
1
vote
0answers
39 views

Limitations to using the method of least squares to fit an ellipse to experimental data?

I have used the method described below many times without issue to determine the major and minor axes of the "data ellipse". However, for this set of data (Fig. A) the method described below is ...
0
votes
1answer
53 views

Computing a best-fit quadratic approximation to a given function over a finite interval

I have a function of the form $$f(b) = \frac{1}{2}(\sqrt{N-b^2}-b+1) $$ on the interval $1\leq b < \sqrt{N/2}$. When I plot this curve for various values of $N$, I get a parabolic shape ($ax^2+bx+c$...
1
vote
0answers
19 views

How can I make a best fit line out of data where each point is weighted more heavily than the previous?

I'm thinking that there should be a linear regression technique that allows me to value recent data more that older data. (Maybe this difference in weight could be configurable through a constant). ...
0
votes
0answers
7 views

Least squares constrained by fixing the average of the predictions

I am unsure whether this question is more suited for mathSE or CrossValidated, but since I am a mathematician I will post it here. Are there any well-known methods to perform a least square ...
0
votes
1answer
43 views

A statistical robust least-squares problem

Consider the following statistical robust least-squares problem $$\text{minimize } \mathbb E \|Ax - b\|_2^2$$ where $A = \bar A + U$, where $\bar A$ is the mean of $A$ and $U$ is a zero-mean ...
0
votes
0answers
8 views

Iteratively Weighted Least Squares and Hessian

The question is about stationary point of a least squares being identical to the stationary point of an error function as stated in: Computer Vision: Algorithms and Applications by Szelinski et al. p....
0
votes
1answer
16 views

Algebraic transformation — where is my mistake?

I tried to find the estimators of $\hat{\beta_1}$ and $\hat{\beta_0}$ via the least-squares method algebraically. Somehow I seem to have messed up. Can you tell me where? My Calculations.
1
vote
0answers
35 views

Linear quadratic regulator via least squares

In this set of slides, the finite horizon LQR problem is stated as a least-squares problem (slide 11), and using a naive method (e.g., QR factorization), the cost to solve this problem is $O(N^3nm^2)$ ...
3
votes
1answer
98 views

Linear Least-Squares Problem with Inequality Constraints on Residual

I have an over-determined linear least-squares problem $$\min_{\vec{x}}\Vert\mathbf{A}\vec{x}-\vec{b}\Vert_2^2,$$ where $\mathbf{A}\in\mathbb{R}^{n\times m}$, $\vec{x}\in\mathbb{R}^m$, $\vec{b}\in\...
0
votes
0answers
11 views

How do I derive a Least Square Estimator of a linear combination of two variables?

I am working on a problem where I have the following model: lm(Y ~ x1 x2) If I have the output of this general model in R, is it possible to derive the LSE of: <...
0
votes
0answers
79 views

Singular matrix C in LSQ subproblem - What does it mean?

I am trying to solve an Inverse Kinematics (IK) task as an optimisation problem using Python and SciPy. There exists a robot arm ...
2
votes
2answers
77 views

Find the derivative of $f(\beta) = (\vec{y} -X\beta)^T(\vec{y} -X\beta)$ using the product rule

So I want to differentiate $f(\beta) = (\vec{y} -X\beta)^T(\vec{y} -X\beta)$ using the product rule. Here: $\vec{y}$ is an $n \times 1$ vector $X$ is an $ n \times p$ matrix $\beta$ is a $p \times 1$ ...
0
votes
2answers
44 views

Least Squares : Approximation of cubic polynomial

I want to determine an approximation of a cubic polynomial that has at the points $$x_0=-2, \ x_1=-1, \ x_2=0 , \ x_3=3, \ x_4=3.5$$ the values $$y_0=-33, \ y_1=-20, \ y_2=-20.1, \ y_3=-4.3 , \ y_4=32....
0
votes
1answer
19 views

Calculating the var(β) in a least square regression model

The linear model that I'm working with is: $$y_t =α +βx_t + ε_t$$ Based on my Lecture I have: $$Var(\hatβ) = Var(Σw_tε_t)$$ where ε is the error term and $$w_t = \frac{x_t-\overline x}{Σ(x_t-\...
2
votes
2answers
49 views

Proving that a given function $f^*$ is the best least square approximation

In De Boor (1972) it is stated that Let be $\$ $ a finite dimensional linear space of functions defined on the interval $[a,b]$. We are searching for the best approximation from $\$$ to $g$. ...
0
votes
1answer
26 views

matrix algebra and idempotent matrix

I'm having a little trouble understanding a few derivations in my book for least squares regression. $\textbf{Question 1}$: If $\textbf{M}^0 \textbf{i} = [\textbf{I} - \frac{1}{n}\textbf{ii'}]\...
0
votes
0answers
38 views

Deriving the least squares estimate of $\beta_{k-1}$

Let $y_i=\Sigma^k_{j=0} x_{ij} \beta_j+\epsilon_i$ $\epsilon_i$ is $NID(0,\sigma^2)$ and $x_{ij}, i=1,...,n, j=0,...,k$ is the $(i,j)^{th}$ elelement of the $n \times (k+1)$ matrix $X$, which is of ...
2
votes
1answer
75 views

How to fit data to a piecewise function?

My question today regards a set of data that I wish to fit a piecewise-defined continuous function. This data set covers a domain of x-values from $0$ to $\mu$ on the x-axis. What I need is to ...
0
votes
0answers
28 views

is this the correct way to solve this equation, to find $W^*$

$$\min_{W} ||XW-X||_F^2+p_1||W||_1+p_2R(W), W\geq0$$ Find derivative of equation above equal to $0$ $$2X^T(XW^*-X)+p_1||W||_1+p_2R(W)=0$$ $$2X^TXW^*-2X^TX+p_1||W||_1+p_2R(W)=0$$ $$2X^TXW^*=2X^TX-p_1|...
0
votes
1answer
25 views

Doubt about the role of “equivalence class of functions” in this Least Square example.

I am dealing with a proof from De Boor (1972) about least square approximations using splines. Suppose we have a set of data and we want to estimate the least square approximation. Let $ \$ $ be a ...
0
votes
0answers
30 views

How to get W* form minimize function

I need help to solve this equation $$\min_{W} ||XW-X||_F^2+p_1||W||_1+p_2R(W), W>=0$$ X size dxn (d = number of feature & n = number of data) Example input: X is random matrix 3x5 $$R(W)=Tr(W^...
0
votes
0answers
37 views

Ols estimator with the errors following a bernoulli distribution

I am having trouble understanding how i should approach the following problem: Given 𝑦𝑖 = 𝛼 + 𝛽𝑥𝑖 + 𝜀𝑖 𝑖 = 1, … , N with 𝜀𝑖 𝑖 = 1,2 … , N being a succession of IID Bernoulli ...
1
vote
0answers
52 views

Higher order lp regularization and lp least squares like regression with p>2

Is $l_p$ norm with $2<p<\infty$ used for regularization in any practical applications? Also least squares is usually used with $l_2$ norm squared. But are there any applications where $l_p$ norm ...
0
votes
0answers
15 views

Difference between solving $[A;B]x = [c;d]$ and ${(A'A)+(B'B)}x = (A'c)+(B'd)$

I have the following minimization problem $$\min{||Ax-c||^2_2+||Bx-d||^2_2}.$$ My question is that how are those solutions in below are different? $$ x = (A^TA+B^TB)^{-1}(A^Tc+B^Td) $$ and $$ x ...
1
vote
1answer
39 views

How to optimize a non-linear least squares energy with respect to the non-zeros of a sparse matrix?

I have an energy I'd like to minimize of the form: $E(G) = \|\underbrace{X - Y G^T L G B}_{f(G)}\|_F^2$ where $X,Y,B$ are dense matrices and $G,L$ are sparse matrices ($G^TLG$ is also sparse), $\|M\...
1
vote
5answers
77 views

Least Squares solution for a symmetric singular matrix

I want to solve this system by Least Squares method:$$\begin{pmatrix}1 & 2 & 3\\\ 2 & 3 & 4 \\\ 3 & 4 & 5 \end{pmatrix}\begin{pmatrix}x\\y\\z\end{pmatrix} =\begin{pmatrix}1\\5\\...
0
votes
0answers
31 views

Least squares and Gram matrix of B-spline derivatives

The Gram matrix of a B-spline basis is defined as $$ G_{ij} = \int_S B_i(x) B_j(x) dx $$ where the integral is taken over the full support $S$ of the B-splines. This matrix is positive definite and so ...
1
vote
1answer
33 views

How to show that the simple least square estimators minimizes the SSE using the second partial derivative test?

I am trying to apply the second partial derivative test to show that the simple least square estimators $\hat\beta_0$ and $\hat\beta_1$ does minimize the sum of the squared errors based on page 3 of ...
0
votes
1answer
23 views

Least Squares Circumcenter of Polygons

It is well known that the circumcenter of a polygon exists if and only if the polygon is cyclic. I would like to extend the definition of an circumcenter for noncyclic polygons. Namely, let us define ...
0
votes
0answers
22 views

Extrapolation error of linear regression lines for a two-cluster data set

I am studying on some linear regression problems using the least-squares method and stumbled upon a problem regarding the error when extrapolating far-away datapoints. About the problem: For a point $...
3
votes
2answers
53 views

Why the average of a set of value has the least square error?

Now we have the equation $$\sum_{i}(x_i-\hat x_i)^2,$$ where $x_i$ is the observed value of a data sample $S$. Here is the question: Why does this expression get its minimum value when $\hat x_i$ ...
0
votes
0answers
39 views

solving large scale and ill-posed least square problem

I want to estimate unknown value using least square. my matrix is very large, dense(full numerical) and ill_conditioned. I have a pc with 128-gigabyte memory, In this system, only the calculations of ...
0
votes
0answers
16 views

2d basis for approximating curvature of spherical surface

I have a surface $h(x,y)=z$ which is point-wise defined and approximately hemispheral. I am using moving least squares to find best-fit function of that surface, and then use known approximation of ...
0
votes
0answers
33 views

Tips for optimisation problem

I have an optimization (minimize) problem which can be written down as: $f(\vec{x})=\sum_1^m{(max(\vec{a_1}*x_1,\vec{a_2}*x_2,\vec{a_3}*x_3,...,\vec{a_n}*x_n)-\vec{a_0})^2}$ Where $m$ is the size of ...
0
votes
1answer
34 views

How can I estimate an first ODE with first order data, with the third order ODE form?

Let's assume that we have the data $y(t), u(t)$ and it's from a first order ODE: $$ \dot y(t) + a y(t) = b u(t) $$ But we have a ODE form at third order: $$ \dddot y(t) + a \ddot y(t) + b \dot y(t) +...
0
votes
0answers
39 views

Numerical analysis: least squares method

Basically I came up with a function where I need to approximate it using the least squares method. I had no trouble computing the constants on MATLAB and got to the following plot: Now as you see the ...
0
votes
0answers
14 views

Eigen characters into least-square method

I want to ask how to proove the $$\widehat{z} $$ by using the eigen characters into least-square method? Q: $$y=Kz$$ where K is a nxn matrix and is a kernel with eigenfunctions $$ \lambda_m \psi_m=K\...
0
votes
1answer
26 views

Derivation of the ordinary least squares estimator β1 and the sampling distribution?

I am trying to derive the ordinary least squares and its sampling distribution for the model: $$y = \beta_0 + \beta_1 x + \epsilon$$ How can I obtain the estimator for $\beta_1$
1
vote
0answers
27 views

Least-squares regularization matrix is not positive definite?

I am fitting data $(x_i,y_i)$ to the following model: $$ f(x) = \sum_j a_j g_j(x) = a^T g(x) $$ where $g_j(x)$ are well-conditioned basis functions (in my application they are B-splines, but I don't ...
0
votes
0answers
21 views

Total Least Square fitting

Say I want to fit a straight line using Total Least Square (as opposed to Least Square), which is to minimize the sum of (yi-k*xi-b)^2/(k^2+1) over all xi's and yi's, where xi's and yi's are training ...
1
vote
1answer
172 views

Norm constrained least square minimization with an additional single linear equality constraint: A quadratically constrained quadratic program (QCQP)

Given is the following QCQP problem: \begin{align} &&\min_{\mathbf{x} \in \mathbb{C}^n} &\|A \mathbf{x} - \mathbf{b}\|^2,\tag{1}\label{1}\\ \text{ subject to:}&&&\\ &&\|...