Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere. See https://en.wikipedia.org/wiki/Least_squares

learn more… | top users | synonyms

0
votes
1answer
11 views

How to shift points optimally for best rounding

I have sets of points. E.g.: 5.664, 2.292, 1.368, 0.18, 3.3, 4.74, 7.812, 6.564, 5.352, 4.008, 2.568, 5.352 I'd like to shift them a bit (add some uniform dx to all of them) to make them closer to ...
1
vote
1answer
33 views

Analytical solution to nonlinear least-squares problem

I have a data set which can be fit well to a single gaussian model, with dependent variables $y_i$ and independent variables $x_i$, with $i=1...N$. I want to avoid using a nonlinear fitting library, ...
0
votes
0answers
4 views

Deriving spectral norm or similar quantity for structured random matrix

I have a problem where I have no idea to start. Suppose a simple Least Squares system with $M$ unknowns $c$ and $N$ observations $y$ which is given through the linear mapping $X$: $$y = X c$$ It is ...
2
votes
1answer
20 views

Properties of the solution of a linear system with random equations

$x_i$ is drawn from $\mathrm{unif}(a,b)$, $y_i$ is drawn from $\mathrm{unif}(c,d)$. $x_i$ are independent from each other. $y_i$ are independent from each other. $x_i$ are independent of $y_i$. $i$ ...
1
vote
0answers
26 views
+50

Does anyone know a reference to best-fitting lines with integral coefficients?

I'm writing up a manual on how to generate "nice" Linear Algebra problems; that is, where the solutions tend to be integral. I "discovered" the following fact about the best-fitting line: Theorem. ...
2
votes
1answer
19 views

Deriving the identity: $\hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}$

For some reason I am having an extremely hard time finding out how the following expression is derived $$ \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} $$ Is ...
0
votes
0answers
14 views

How to use leave-one-out cross-validation scheme to compute the accuracy of a linear model fit

Using the least squares estimation I calculated the model fit for a dataset where: $$ p = \beta_{0} + \beta_{1} * t $$ How could I use leave one out cross-validation(CV) scheme to compute accuracy ...
0
votes
0answers
20 views

Least-squares when some coefficient is $0$?

I'm trying to find least squares approximation $p(x)=c_1x+c_2x^2$ of $f(x)=xe^{x/2}$ in $[0,2]$. Using the algorithm here, p.7.: http://www.math.niu.edu/~dattab/MATH435.2013/APPROXIMATION.pdf I'm ...
1
vote
1answer
19 views

Normal system of the least square method

I'm trying to show the following. $Pa$ is the approximation system of $y$. I want to show that finding the minimmum for the function $$f(a,y)=||Pa-y||_2^2$$ is equivalent to solve the normal system of ...
0
votes
0answers
11 views

Inverse kinematics - How do i compute the du?

I am at the moment trying to implement at jacobian based inverse kinematics solver, which is given a current homogeneous Transformation matrix r(q) and a desired homogenous tranformation matrix ...
0
votes
0answers
16 views

Does this method for solving an overdetermined systemof equations minimize an error?

Suppose we have an overdetermined set of linear equations $X\beta=y$, which we wish to solve for $\beta$. $X$ is an $n\times m$ matrix with $n>m$. The standard method of doing this is, of course, ...
0
votes
0answers
25 views

Image processing, optimization via regularization - efficient strategy

i would like to solve the following system: $J(x) = |Ax-b|_2^2+\gamma|\nabla x|_2^2$ subject to: $x \geq 0, \sum_i x_i = 1$ The underlying problem is to derive the PSF from a sharp and blurry ...
0
votes
0answers
22 views

Number of measurements for least squares and relation to maximum likelihood

I have a simple overdetermined system of equations: $$ y = Xc + e $$ $y, e \in \mathbb{R}^n$, $c \in \mathbb{R}^m$, $X \in \mathbb{R}^{n \times m}$, $e \sim \mathcal{N}(o,\sigma^2)$, $n>>m$ ...
1
vote
1answer
21 views

Understanding Linear Least Squares Problem Statement

My computer vision class is covering linear least squares problems with SVD. In our notes, the problem statement is as follows: min $\mid Ax - b\mid^2 = min(Ax - b)^T(Ax - b)$ = min $x^TA^TAx - ...
1
vote
2answers
14 views

How to simplify OLS formulas?

Let $\lbrace x_i,y_i\rbrace_{i=1}^n$ be a random sample. I am trying to simplify the following expression $$\frac{\sum_{i=1}^n x_i y_i - n \bar{X}\bar{Y}}{\sum_{i=1}^n x_i^2 -n\bar{X}^2}$$ to show ...
1
vote
1answer
24 views

Solving Norm-Constrained Homogeneous Linear Least Squares

I am learning how to solve a norm-constrained homogeneous linear least squares problem. min $(norm(Ax))^2$ for x such that norm(x) = 1 The problem is set up with a Lagrangian as follows: cost = ...
0
votes
0answers
29 views

Residual norm of active set method in non negative least squares

I am having trouble with understanding one of the statements in active set methods done for non negative least squares given in Book written by Lawson. The quadratic problem can be written as ...
0
votes
1answer
44 views

How to plot correct best fit line?

I'm working on a software where I'm plotting graphs and finding best fit line. I have used Least-Square Method and linear regression technique with y = mx + c My problem is that when most of the X ...
0
votes
0answers
32 views

Least Squares with Singular $AA^T$

Given the following system, find all least squares solutions: $\begin{bmatrix}1 & 2 & 3\\2 & 3 & 4\\3 & 4 & 5\end{bmatrix} \vec{x} = \begin{bmatrix}1\\1\\2\end{bmatrix}$ ...
0
votes
0answers
15 views

Solving non-linear functional equations numerically by sequence of linear least-squares?

So I am experimenting with a linear systems solver to find new exciting applications for it. While it is possible to play around to solve some of the more basic functional equations, I am trying to be ...
1
vote
0answers
18 views

Regularization least squares

Given image $x$ and a transformation (blurring) $K$, we get a blurry image $f$. The blurring transformation $K$ is ill conditioned. For a given $f$, find $x$. Solution via regularization: ...
0
votes
1answer
22 views

Fitting a curve of the form Y = ae^(-b) using Least Squares Estimators

I'm trying to fit a set of data into the curve Y=ae^(-bx) There are 6 pairs of data (time,quantity). I have used log on both sides, which gave me ...
0
votes
0answers
10 views

Different ways of solving $\underset{\mathbf{s}}{\text{min}}\;\|F\mathbf{s}-\mathbf{x}\|_{l_2}^2 + \|W\mathbf{s}\|_{l_2}^2$ least square problem?

The problem that I am going to describe arises from compressed sensing technique and after using weighted least squares it can be transformed into the following least squares problem: ...
0
votes
1answer
27 views

Assumptions required for expected value of sum of products to equal zero.

First define $x_i$ is determenistic variable, $v_i$ is random variable. Consider the expression: $$\text E(\sum_{i=1}^{\ n} x_iv_i)= \sum_{i=1}^{\ n} \text E (x_iv_i)=\sum_{i=1}^{\ n} \text E (x_i) ...
1
vote
1answer
30 views

How can I force least square solution matrix to be diagonal?

Let's say I have the following equation $$AX=B$$ where $A$ is a $8\times 3$ matrix (known), $X$ is a $3\times3$ "diagonal" matrix which represents the coefficients (unknown) and $B$ is a ...
0
votes
0answers
19 views

general framework for proof of convexity in least square problem

Let $f(x_i,\theta) : X \times \Theta \rightarrow Z$ be a parametric function with parameter $\theta$ that we wish to fit to set of samples $S =\{(x_i,z_i)\}_{i=1}^N$, where $(x_i, z_i) \in X \times ...
1
vote
1answer
61 views

why do we say SVD can handle singular matrx when doing least square? Comparison of SVD and QR decomposition

I don't quite understand why we say that QR decomposition doesn't handle singular matrix, while SVD does when they are used for least square problem? My example in Matlab seems to support the ...
1
vote
0answers
33 views

How to constrain the linear least squares fit of a quadratic polynomial with known constraints

How to constrain this fit I have some function , $f(x) = a x^2+b x+c$ , with the constraints $a<0$ and $c = \frac{b^2}{4a}+\frac{1}{2}ln(\frac{-a}{\pi})$ I have measured $f(x)$ for some $x$. Can ...
1
vote
0answers
11 views

regularized least squares (L1 norm)

My objective function that is to be minimized is as follows: $$\|y-Ax\|_2^2 + \alpha\|Lx\|_1$$ where $L$ is the gradient operator. Now this problem seems convex because the first term is quadratic ...
3
votes
1answer
43 views

Power curve fitting

Least Squares can be used to fit the following power curve to given data points. $y=ax^b$ where $a,b$ are constants to be determined by the fitting process as seen here. Is there a way to fit ...
1
vote
1answer
29 views

The validity of Least Squares Fitting to a specific problem

I'm going to try and keep this question broad, so I apologise if it's poorly written. I have a series of functions; $$ \Psi_{j} = \sum_{n = 1}^{N} A_{n} \sinh{2 \pi n S_{j}} \cos{2 \pi n X_{j}} + ...
0
votes
0answers
29 views

Least square prediction using covariance and autocorrelation

I have a data sequence $x = 1,1,2,3,5,8,13$ Now, I have the following linear predictor: $x(t)=a_1x(t-1)+a_2x(t-2)+e(t)$ where e[t] is the prediction error Determine the least square coefficients ...
0
votes
0answers
6 views

Comparing goodness of fit 2 samples different variance

I'm fitting a model to two samples of data, linear least squares minimizing the chi squared. The samples are slightly different, one has much larger variance than the other so will naturally have a ...
0
votes
0answers
56 views

Is a least squares solution to $Ax=b$ necessarily unique

Let $A$ be an $m$ x $n$ matrix, and suppose that $b\in\mathbb{R}^n$ is a vector that lies in the column space of $A$. Is a least squares solution to $Ax=b$ necessarily unique? If so, give a detailed ...
0
votes
1answer
24 views

least-squares estimation

I want to write a computer program that findes the least-squares estimates of the coefficients in the following models: 1) $y = ax^2+bx+c$ 2) $y= ax^n$ can you help me what should I do?I don't have ...
1
vote
0answers
24 views

Does Least Squares Regression Minimize RMS

In the application of least squares regression to data fitting, the quantity of minimization is the sum of squares (sum of squared errors, to be specific). I believe this fitting also minimizes the ...
1
vote
1answer
29 views

About R-squared

Let $x$ and $y$ vectors $n\times1$ and $\hat{\beta}_{xy}$, $\hat{\beta}_{yx}$ the coefficients obtained by OLS of the regressions $x$ over $y$ and $y$ over $x$ respectively. Show that ...
0
votes
1answer
23 views

About Linear Regression

Let the regression model of $y_i$ over $x_i$, $i=1,2...,N$$$y_i=\beta x_i+u_i$$, then $$\beta=\frac{\sum y_ix_i }{\sum x_i^2}=\frac{\sum y_i }{N}=\bar{y}$$ The problem is that I can't see how to ...
2
votes
2answers
34 views

Least squares regression with two predictor variables (exponential functions of time)

Question cropped from textbook (Apologies for the link- I don't have enough rep to post the actual image.) [Now pasted below. Ed.] I've come across a question in a textbook (linked above) requiring ...
-1
votes
0answers
14 views

Two exercises from linear regression

Click for image These are the two tasks I have been thinking about for two much time. How should I even start? Vectors are not my strong side. Any help is welcomed.
1
vote
1answer
33 views

Application of weighted least squares to a log linear equation

I am trying to fit a curve to a set of data using a weighted least squares approach. The reason I am using the weighted approach is to bias my solution to my more reliable data. I am however having a ...
0
votes
3answers
49 views

How did he get this outcome?

It's a matrix solved with least squares equations (probaly). I used some calculator but can't get his outcome. If you have a way how to get to this please explain how. [The math on that image is: $$A ...
3
votes
1answer
62 views

How do you prove that if $x$ is such that $A^T(b - Ax)=0$ then it minimizes $\| Ax - b \|^2$ without calculus?

I know this is is a well known fact linear algebra (and feel should be super obvious) but I got a little stuck. I was to prove that choosing an $x \in \mathbb{R}^K$ such that $A^T(b - Ax) = 0$ is ...
0
votes
0answers
23 views

Maximum Likelihood Estimator of exponential of L2 norm

given the observed data $x = (x_1; x_2; \cdots; x_n)^T$ , the likelihood function p(x; $\theta$) can be charaterized as $$p(x; \theta) = \alpha(x) e^{ ||x - \hat{\theta}||_2} $$ where $\hat{\theta} ...
0
votes
0answers
43 views

maximum likelihood estimation of exponential and polynomial components model

I tried to find the maximum likelihood estimator and MMSE of the non linear model but I got stuck. Can you help me to explain it? The output of a system can be modeled using a combination of ...
2
votes
0answers
38 views

Why subtract mean and divide by standard deviation?

MATLAB's polyfit function "finds the coefficients of a polynomial P(X) of degree N that fits the data Y best in a least-squares sense." Prior to fitting, the function scales the independent variable, ...
3
votes
1answer
34 views

Least squares with multiple linear constraints

The method of direct elimination can be used to solve the constrained least squares problem \begin{equation} \min_{\mathbf{x}}\left\Vert \mathbf{Ax}-\mathbf{b}\right\Vert _{2} \end{equation} ...
2
votes
1answer
41 views

Least Squares Fitting Quadratic Equation to a set of points

My math skills from my college days are a bit rusty, so if my terminology is wrong, I apologize. I will try to be as clear as I can. I have a set of 50 points on a plane that roughly follow the ...
0
votes
1answer
28 views

Need a symbolic solution to least-squares optimization problem

What's the symbolic solution to this problem? Minimize $\left(x-x_T\right){}^2+\left(y-y_T\right){}^2+\left(z-z_T\right){}^2$ for variables $x,y$ and $z\in \mathbb{R}$, given parameters $x_T, y_T, ...
1
vote
0answers
18 views

Can the Moore-Penrose conditions be compressed?

For the $m * n$ matrix $A$ the matrix $A^{+}$ is called its Moore-Penrose pseudo inverse, if for $A^{+}$ the Moore-Penrose conditions hold. The Moore-Penrose conditions are the following: ...