Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

2
votes
1answer
48 views

How to fit data to a piecewise function?

My question today regards a set of data that I wish to fit a piecewise-defined continuous function. This data set covers a domain of x-values from $0$ to $\mu$ on the x-axis. What I need is to ...
0
votes
0answers
20 views

is this the correct way to solve this equation, to find $W^*$

$$\min_{W} ||XW-X||_F^2+p_1||W||_1+p_2R(W), W\geq0$$ Find derivative of equation above equal to $0$ $$2X^T(XW^*-X)+p_1||W||_1+p_2R(W)=0$$ $$2X^TXW^*-2X^TX+p_1||W||_1+p_2R(W)=0$$ $$2X^TXW^*=2X^TX-p_1|...
0
votes
1answer
18 views

Doubt about the role of “equivalence class of functions” in this Least Square example.

I am dealing with a proof from De Boor (1972) about least square approximations using splines. Suppose we have a set of data and we want to estimate the least square approximation. Let $ \$ $ be a ...
0
votes
0answers
27 views

How to get W* form minimize function

I need help to solve this equation $$\min_{W} ||XW-X||_F^2+p_1||W||_1+p_2R(W), W>=0$$ X size dxn (d = number of feature & n = number of data) Example input: X is random matrix 3x5 $$R(W)=Tr(W^...
0
votes
0answers
19 views

Ols estimator with the errors following a bernoulli distribution

I am having trouble understanding how i should approach the following problem: Given 𝑦𝑖 = 𝛼 + 𝛽𝑥𝑖 + 𝜀𝑖 𝑖 = 1, … , N with 𝜀𝑖 𝑖 = 1,2 … , N being a succession of IID Bernoulli ...
1
vote
0answers
46 views

Higher order lp regularization and lp least squares like regression with p>2

Is $l_p$ norm with $2<p<\infty$ used for regularization in any practical applications? Also least squares is usually used with $l_2$ norm squared. But are there any applications where $l_p$ norm ...
0
votes
0answers
11 views

Difference between solving $[A;B]x = [c;d]$ and ${(A'A)+(B'B)}x = (A'c)+(B'd)$

I have the following minimization problem $$\min{||Ax-c||^2_2+||Bx-d||^2_2}.$$ My question is that how are those solutions in below are different? $$ x = (A^TA+B^TB)^{-1}(A^Tc+B^Td) $$ and $$ x ...
1
vote
1answer
33 views

How to optimize a non-linear least squares energy with respect to the non-zeros of a sparse matrix?

I have an energy I'd like to minimize of the form: $E(G) = \|\underbrace{X - Y G^T L G B}_{f(G)}\|_F^2$ where $X,Y,B$ are dense matrices and $G,L$ are sparse matrices ($G^TLG$ is also sparse), $\|M\...
1
vote
5answers
66 views

Least Squares solution for a symmetric singular matrix

I want to solve this system by Least Squares method:$$\begin{pmatrix}1 & 2 & 3\\\ 2 & 3 & 4 \\\ 3 & 4 & 5 \end{pmatrix}\begin{pmatrix}x\\y\\z\end{pmatrix} =\begin{pmatrix}1\\5\\...
0
votes
0answers
19 views

Least squares and Gram matrix of B-spline derivatives

The Gram matrix of a B-spline basis is defined as $$ G_{ij} = \int_S B_i(x) B_j(x) dx $$ where the integral is taken over the full support $S$ of the B-splines. This matrix is positive definite and so ...
1
vote
1answer
26 views

How to show that the simple least square estimators minimizes the SSE using the second partial derivative test?

I am trying to apply the second partial derivative test to show that the simple least square estimators $\hat\beta_0$ and $\hat\beta_1$ does minimize the sum of the squared errors based on page 3 of ...
0
votes
1answer
22 views

Least Squares Circumcenter of Polygons

It is well known that the circumcenter of a polygon exists if and only if the polygon is cyclic. I would like to extend the definition of an circumcenter for noncyclic polygons. Namely, let us define ...
0
votes
0answers
17 views

Extrapolation error of linear regression lines for a two-cluster data set

I am studying on some linear regression problems using the least-squares method and stumbled upon a problem regarding the error when extrapolating far-away datapoints. About the problem: For a point $...
3
votes
2answers
52 views

Why the average of a set of value has the least square error?

Now we have the equation $$\sum_{i}(x_i-\hat x_i)^2,$$ where $x_i$ is the observed value of a data sample $S$. Here is the question: Why does this expression get its minimum value when $\hat x_i$ ...
0
votes
0answers
34 views

solving large scale and ill-posed least square problem

I want to estimate unknown value using least square. my matrix is very large, dense(full numerical) and ill_conditioned. I have a pc with 128-gigabyte memory, In this system, only the calculations of ...
0
votes
0answers
13 views

2d basis for approximating curvature of spherical surface

I have a surface $h(x,y)=z$ which is point-wise defined and approximately hemispheral. I am using moving least squares to find best-fit function of that surface, and then use known approximation of ...
0
votes
0answers
30 views

Tips for optimisation problem

I have an optimization (minimize) problem which can be written down as: $f(\vec{x})=\sum_1^m{(max(\vec{a_1}*x_1,\vec{a_2}*x_2,\vec{a_3}*x_3,...,\vec{a_n}*x_n)-\vec{a_0})^2}$ Where $m$ is the size of ...
0
votes
1answer
30 views

How can I estimate an first ODE with first order data, with the third order ODE form?

Let's assume that we have the data $y(t), u(t)$ and it's from a first order ODE: $$ \dot y(t) + a y(t) = b u(t) $$ But we have a ODE form at third order: $$ \dddot y(t) + a \ddot y(t) + b \dot y(t) +...
0
votes
0answers
38 views

Numerical analysis: least squares method

Basically I came up with a function where I need to approximate it using the least squares method. I had no trouble computing the constants on MATLAB and got to the following plot: Now as you see the ...
0
votes
0answers
12 views

Eigen characters into least-square method

I want to ask how to proove the $$\widehat{z} $$ by using the eigen characters into least-square method? Q: $$y=Kz$$ where K is a nxn matrix and is a kernel with eigenfunctions $$ \lambda_m \psi_m=K\...
0
votes
0answers
17 views

What is the largest condition number a 64-bit computer can take to do matrix inversion to give good result?

In my problem, it looks like $10^{13}$ is a red-line, once it crosses, the performance of matrix inversion goes down. But why? Or do you have better idea for that?
0
votes
1answer
19 views

Derivation of the ordinary least squares estimator β1 and the sampling distribution?

I am trying to derive the ordinary least squares and its sampling distribution for the model: $$y = \beta_0 + \beta_1 x + \epsilon$$ How can I obtain the estimator for $\beta_1$
1
vote
0answers
17 views

Least-squares regularization matrix is not positive definite?

I am fitting data $(x_i,y_i)$ to the following model: $$ f(x) = \sum_j a_j g_j(x) = a^T g(x) $$ where $g_j(x)$ are well-conditioned basis functions (in my application they are B-splines, but I don't ...
0
votes
0answers
19 views

Total Least Square fitting

Say I want to fit a straight line using Total Least Square (as opposed to Least Square), which is to minimize the sum of (yi-k*xi-b)^2/(k^2+1) over all xi's and yi's, where xi's and yi's are training ...
1
vote
1answer
142 views

Norm constrained least square minimization with an additional single linear equality constraint: A quadratically constrained quadratic program (QCQP)

Given is the following QCQP problem: \begin{align} &&\min_{\mathbf{x} \in \mathbb{C}^n} &\|A \mathbf{x} - \mathbf{b}\|^2,\tag{1}\label{1}\\ \text{ subject to:}&&&\\ &&\|...
0
votes
0answers
9 views

Why residuals are a good estimator of random disturbance?

Let a linear OLS model: $$Y= X \beta + u$$ Where $u$ is a random disturbance. If we define the residual of the regression as $$e = Y - X \widehat{\beta}$$ where $\widehat{\beta}$ is the OLS vector of ...
0
votes
0answers
21 views

How do you find two curves that enclose a set of data points?

I'd assume you'd use least squares here to first get the best fit line for the data, but I need to specifically minimize the vertical distance between the two curves. Originally, I was thinking I'd ...
2
votes
1answer
55 views

why $x = \mathbf A^{\dagger}b$ is the one that minimizes $|x|$ among all mimizers of $|\mathbf Ax - b|$

for arbitrary matrix $\mathbf A\in \mathbb R^{m \times n}$ and $rank(\mathbf A) = r$, solve the least squares: $$\min \|\mathbf Ax - b\|_2. $$ According to SVD, pseudo inverse of $\mathbf A$ is $$\...
0
votes
0answers
20 views

Norm of least squares residual

Let $A=QR$ be the full $QR$ factorization of $A\in\mathbb{C}^{m\times n}$, $m\ge n$, where $$Q=\begin{bmatrix} \hat{Q}_1 & \hat{Q}_2 \end{bmatrix}, R=\begin{bmatrix} \hat{R}_1\\ 0 \end{bmatrix},$...
2
votes
2answers
38 views

Confused about the terminology of least regression squares

I am confused by the language of my math text book where it says $\hat y = \alpha \times$basis function summation from $i=1$ to $n$? What are basis functions? And why does it says there are more data ...
0
votes
1answer
34 views

Im confused with Least Squares Regression Derivation (Linear Algebra)

I am having troubles understanding Least Squares Regression Derivation (Linear Algebra) in order to code it in matlab. As far as I have understood, is that you take the residual value = (yhat - y)^2 . ...
0
votes
0answers
12 views

Weighted rank 1 approximation to matrix

I want to solve the following problem: $$\arg\min_{u,v} \|W\odot(u v^\mathsf{T}-M)\|_\mathrm{F}$$ where $u$ and $v$ are $N\times 1$ vectors, $W$ and $M$ are $N\times N$ matrices, $\odot$ represents ...
0
votes
1answer
36 views

Least Squares Solution of Minimal Norm when $A^{*}b = 0$

Suppose, given a matrix $\textbf{A} \in \mathbb{C}^{m \times n}$ and a vector $\textbf{b} \in \mathbb{C}^{n}$, I want to find the minimal norm solution of $$\min_{\textbf{x}}\|\textbf{A}\textbf{x} - \...
0
votes
0answers
26 views

Proof that if $x$ minimises $\left \| Ax-b \right \|^{2}_2 +\mu\left \| x \right \|^{2}_2$ then it solves regularised normal equations

I'm looking for a proof that if $x$ minimises $\left \| Ax-b \right \|^{2}_2 +\mu\left \| x \right \|^{2}_2$ then it solves $(A^{T}A+\mu I)x=A^{T}b$
0
votes
0answers
22 views

Is it possible to get a least squares for the Uniform distribution?

Is it possible to use a Least Square Method for the Uniform distribution? That is, for U~[a,b] is it possible to get LS estimators of a and b? If not, what is the reason?
0
votes
1answer
25 views

fractional curve fitting of the function $y=a+bx^{\alpha}$

Assume I have a set of data $(x_{i},y_{i})$, $i=1,...,m$. How can we find the best values of the parameters $a$ and $b$ and $\alpha$ such that the curve $y=a+bx^{\alpha}$ best fits the data. This ...
0
votes
1answer
24 views

How to find a matrix closest to a given matrix in a Inner product space?

Consider $M_2(\mathbb{C})$ with the inner product $$\langle A, B \rangle = trace(B^*A)$$ where $*$ is conjugate transpose. Find the closest element of the complex symmetric $2\times 2$ matrices to $$A ...
0
votes
0answers
23 views

Low rank approximation or Matrix Factorization?

Hi,In this question ,I need to use low rank approximation or matrix factorizaton?As I see, A is an orthogonal matrix since A^T*A-I
0
votes
1answer
13 views

Why there are n-r dimensional set of vectors x for rank-deficient least square problem

I saw this theorem on the lecture slides (http://www2.aueb.gr/users/douros/docs_master/Least_Square_pr.pdf) with topic Rank-Deficient Least Squares Problem Given $A\in R^{m\times n}, m > n, \, r ...
0
votes
0answers
20 views

Multi-Linear regression to find a symmetric matrix

I am trying to solve a multiple linear regression problem in the form of $y = A \ x$ where $y,A,x \in \mathbb{C}$ with unknown $A$. This may be reasonably easy by using the normal equation for ...
0
votes
0answers
28 views

Best fit line with known offsets.

Hello all it's my first post on these forums so if I'm breaking any ettiqute feel free to let me know. So it's been a few years since I took a linear algebra class in college but I remember using ...
3
votes
1answer
113 views

Optimize $M$ such that $MM^T$ is the “smallest” (in a positive semi-definite sense)

I want to solve the following problem: \begin{alignat}{} \underset{M}{\text{minimize}} \quad & MM^T \\ \text{subject to} \quad & MF=I. \end{alignat} where by minimize $MM^T$ I mean to find $...
0
votes
1answer
21 views

Least square solution to the system

I am trying to solve the following problem: Let $u_1$ and $u_2$ be two orthogonal vectors in ${\rm I\!R}^n$ and set $a_1 = u_1$, $a_2 = u_1 + \varepsilon u_2$ for $\varepsilon>0$. Let also $...
1
vote
0answers
19 views

Nonlinear least squares with analytical solution

I want to find a "true" nonlinear least squares problem which does have an analytical solution. I tried to construct something with a Dirac-Delta function and ended up with $y_n = c^2\delta(x_n-x_1)...
0
votes
1answer
21 views

How is multiplying A by A transpose related to the gradient in the least squares problem?

I have a function: $f(\textbf{x}) = \frac{1}{2} || \textbf{Ax - b} ||_2^2$ I am trying to minimize this function on values of x using gradient based optimization. The textbook I am following ...
0
votes
1answer
53 views

Sparse recovery with L1 shrinkage iteration for higher denominational image classification

For 2 months I have been studying sparse recovery and its applications for image classification and I have found that it's a broad area in mathematics which gives rise to a wide variety of ...
0
votes
0answers
24 views

When can Levenberg-Marquardt fitting algorithm be used with least absolute residuals (LAR) method and not Bisquare method for residual minimization?

I am sorry for asking a trivial question but though I have found an answer in the following link, I would like to know some more insight on the situations when one residual minimizing method is used ...
0
votes
0answers
26 views

Time complexity of least squares curve optimization using QR decomposition with Householder method.

Given a set of $m$ pairs of points, $<x_k, y_k>$, and a curve $y=ax^4 + bx^2 +c $ use the least squares method with QR decomposition and the Householder algorithm to approximate the curve's ...
0
votes
1answer
26 views

Distributional assumptions in Maximum likelihood estimator (MLE) and least squares estimator (LSE)

Many textbooks do mention that MLE does need some distributional assumptions but I could never find which they are. LSE on the opposite, doesn't need distributional assumptions but when missing ...
0
votes
2answers
64 views

Why does solving $A^{\rm T}Ax = A^{\rm T}b$ yield a least squares approximation?

I was following a linear algebra course, and I came upon an example where a linear regression was done by solving $A^{\rm T}Ax = A^{\rm T}b$, where $Ax = b$ could not be solved because $b$ is not in ...