Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere. See https://en.wikipedia.org/wiki/Least_squares

learn more… | top users | synonyms

0
votes
0answers
32 views

Equivalence of two linear least squares problem

Here I want to build a subspace representation $Uq$ to approximate $x$, where $q$ is the reduced coordinates. We know that the best approximation to $x$ is the linear least squares solution $q_1 = ...
0
votes
0answers
17 views

What is my error in this matrix / least squares derivation?

I'm doing a simple problem in linear algebra. It is clear that I have done something wrong, but I honestly can't see what it is. let, $y = Ax$, $y_{ls} = Ax_{ls}$ where A is skinny, and $x_{ls} = ...
1
vote
0answers
28 views

Minimizing Sum of Least Squares in Matlab

I am working on this minimization problem for image warping that I want to solve in Matlab: Each feature $p$ can be presented by a 2D bilinear interpolation of the four vertices $V_p = [v_p^1, ...
0
votes
0answers
46 views

Pseudo-inverse gives the minimum norm for x

So the pseudo-inverse gives the solution $x = A^+ b$ which minimizes $||Ax - b||_2$. How do I prove that $x$ also has the smallest 2-norm for all $x_i$ where $||Ax - b||_2 = ||Ax_i - b||_2$?
0
votes
1answer
16 views

Regarding least squares, value of n in a scatter plot

I am currently in a college algebra class wherein I am required to do a rather lengthy project regarding least squares. One particular exercise posits the following (keep in mind that in this project ...
0
votes
0answers
24 views

How to solve the least square with $L_2$ norm constraint directly?

I answered the question Why are additional constraint and penalty term equivalent in ridge regression? earlier, but I myself still have some questions on it. To solve \begin{align} \min_{\beta} ...
0
votes
1answer
31 views

Find a line such that sum of perpendicular distances of points to the line is minimized

Given a set of points (column vectors) $S = \{p_1, p_2, \cdots, p_n\} \subset \Re^d$, let $A \in \Re^{n \times d}$ be a matrix of which each row is just $p_i^T$. It is easy to find a unit vector $s_1$ ...
1
vote
0answers
24 views

Increase the probability of correct prediction using multiple regression

First off let me begin by saying that I'm brand new to statistics and I would appreciate it if you could dumb down any answers for my problem. I am trying to create a general prediction of how much a ...
0
votes
1answer
23 views

How to approach this least square projection question?

A simple linear regression model as follows, \begin{align} Y=\beta_0+\beta_1 X+\epsilon \end{align} Now I would like to replace $X$ with another variable $Z$. I only know $X$ and $Z$ are correlated ...
0
votes
0answers
23 views

Motivation for gradient descent method over OLS/MLE for simple linear regression?

I am beginner in machine learning and I am currently trying to find the motivation for gradient descent method. I am confused why we want to employ gradient descent method for linear regression? I see ...
0
votes
1answer
22 views

Does $\vec{b}$ have to be in $\text{im }(A)$ if $|| \vec{b}-A\vec{x}^*||=0$?

I'm going through a least squares computation where $A=\begin{bmatrix}3&2\\5&3\\4&5\end{bmatrix}$ and $\vec{b}=\begin{bmatrix}5\\9\\2\end{bmatrix}$. From ...
0
votes
2answers
48 views

What are some real life applications of least squares problem?

I'm looking for some applications that require solving the least square problem. I know polynomial fitting is one of them, but sure there are many others. Thanks
1
vote
4answers
40 views

Finding initial values with the help of least square method

$D: \begin{bmatrix}1&4.19\\2&3.40\\3&2.80\\4&2.30\\5&1.99\\6&1.70\\7&1.51\\8&1.34\\9&1.21\\10&1.09\end{bmatrix}$ In this table of data the first column is ...
0
votes
1answer
26 views

Least squares fit to a an exponential equation with one unknown

I have this equation $$y = s - cx^{1.85}$$ where s is a known integer and c is unknown. I want to use the least squares method to find the best value of c that fits a set of points. I've used ...
1
vote
1answer
24 views

Least squares approximation to a subspace.

Consider the inner product space $C[0,1]$ with inner product $$\langle f,g\rangle =\int_0^1f(x)g(x)\,dx$$ Let $S$ be the subspace spanned by $1$ and $2x-1$ Find the best least squares approximation ...
0
votes
0answers
8 views

A least squares subject to orthogonal constraint

A least square minimization subject to orthogonal constraint: Assume we are given two matrices $B$ and $A$, we aim to find the following $X$, $\min_A \|B - XA \|_F^2, \mbox{subject to} ~XX^T=I$, ...
0
votes
0answers
13 views

marginal return as a function of full return and inverse diagonal

I met this formula somewhere in our system, and even remember that I proved something like that a lot time ago. Now I became older and probably my mind is not so flexible as before,so I am confused. ...
0
votes
0answers
15 views

algorithms for constrained linear least-squared problems

Given $x_\text{opt}, b\in \mathbb{R}^n$, $A\in\mathbb{R}^{n\times n}$, I'm looking for $x\in \mathbb{R}^n$ such that $$ \min_x \|x - x_\text{opt}\|_2^2,\\ A x \le b,\\ x \ge 0. $$ Apparently, this ...
0
votes
0answers
15 views

curve fittin with non-gaussian noise

Fitting with the least squares method results in the ML fit assuming the given points have a gaussian distributed noise. What methods are there for non-gaussian noise distributions, especially ...
1
vote
0answers
10 views

3D topographic progress compensation by the least squares method.

I'm looking for an explanation of the least squares method used in the case of a correction of 3D point network. We have reference points with known coordinates XYZ, we calculate intermediate points ...
0
votes
0answers
19 views

Express the estimation of the difference of means between two grooups under equal variance using a linear model.

Using a linear model, I've constructed two variables $y$ and $z$: $y = \alpha + \beta x + \epsilon $ $z = \alpha' +\beta'x + \epsilon' $ and I am assuming the difference of the two variables can be ...
0
votes
3answers
94 views

Solving $Ax=b$ when $x$ and $b$ are given.

I am trying to study least square and linear regression and I understand the solution for $Ax = b$ when x is the unknown and the LS solution is given by $(A^TA)^{-1}A^TA$. Now, I was wondering if ...
2
votes
1answer
27 views

why conventional approximation method is true?

why the text book method for finding the fitting curve is right ? we have n data we want to approximate with a polynomial of degree m $P_m(x)$ (m < n-1). and of course $E = \sum_{i=1}^m ...
0
votes
0answers
15 views

Measure best fitting major and minor axis length given 3 points on an ellipse

I am trying to measure the parameters of an ellipse in an image. I have the center, the rotation of the ellipse. I am trying to find the best fitting major and minor axis length based on 3 given ...
0
votes
0answers
9 views

Residual error between two transformations

Suppose I have two unknown 3D transformations matrices in homogeneous coordinates: X and Y. I want to calculate both of these by ...
0
votes
0answers
26 views

How to restrict an angle to a particular interval during iterative least squares inversion?

I have some data and I want to fit a model to it which is a function of 3 orthonormal eigenvectors, using damped iterative least squares inversion. My orthonormal eigenvectors A, B, and C are ...
1
vote
0answers
40 views

Least-squares solution to a transformation between coordinate frames

Suppose I have four coordinate frames in 3D space: A, B, X and ...
0
votes
0answers
23 views

Linear Squares for a system $Ax=b$, multiplying rows

Let multiply the $i$-th row of $A$ by a scalar $\alpha\ne 0$. Now multiply the $i$-th entry of $b$ by the same amount. The solution of the Euler Equation $A*Ax=A*b$ (where $A$ and $b$ are modified) is ...
0
votes
1answer
97 views

The $\alpha$ estimation for the model $x_i = \xi_i \cdot \alpha$

We have $n$ sensors $X_i$ which estimate the scalar value $\alpha$ with different relative accuracies $\delta_i \ll 1$: $$ x_i = X_i(\alpha) = \xi_i \cdot \alpha, \ \ \ \xi_i \sim N(1, \delta_i) $$ ...
0
votes
1answer
34 views

Symmetric matrix-Spectral theorem

Assume we have a matrix $A$ let's say $100 \times 4$. We determine the product $B=A^{T}A$ Then by the spectral theorem \begin{equation} B =U^{T} \lambda U \end{equation} $B$ is a symmetric matrix ...
0
votes
0answers
11 views

Compute the baseline predictor based on the matrix

I need to compute the baseline predictor for the following matrix using a least squares with sixteen equations and nine variables and I have no idea how to do this.
0
votes
0answers
25 views

Modeling Question: Finding a Fit for the form W=kl(g^2)

I have a modeling question on my assignment that I am unsure about. I am given a set of points $W, l,$ and $g$. I have to find some $k$ for the data to optimally fit $W=klg^2$. At first, I tried ...
0
votes
0answers
24 views

Nonlinear least squares problems with binary variables

I want to solve the heat equation $T_t(x,t) = - L_x . T(x,t) + F(x,t)$ in an edge-weighted graph where $L_x = \sum_i x_i e_{ij}$ is weighted Laplacian matrix of the graph. Then I conclude to the ...
0
votes
0answers
15 views

Best size data set for trend line estimation

If I have a data set of varying size for prices of an item over time, and I want to attempt to predict future prices a week into the future, how much of the data set should I be using for my ...
0
votes
2answers
63 views

Least squares minimization of point distances (nonlinear)

We have two sets of 2D points $\bar{x}\leftrightarrow \bar{x}'$ (the bar denotes a vector, i.e. $\bar{x}=(x,y)^{T}$). I would like to minimize discrepancy between the points using the least squares ...
0
votes
0answers
11 views

Optimum delay of complex signals for LS minimization

Say I have a complex periodic signal $s(t)$ with period $T$. Note that $s(t)$ is not necessarily a complex sinusoide. Now I define $h(t)$ as \begin{equation} h(t) = \sum_{i}^{N} s(t - \tau_{i} ) ...
0
votes
0answers
21 views

PCA geometric interpretation

I'm trying to understand what is the geometric interpretation of PCA, but I cant figure out which line will PCA produce
1
vote
0answers
20 views

Solving a sparse Least Squares problem or breaking it to separate least squares problems

I would like to solve a Least Squares (LS) problem of the form y=Fx. Let's assume for simplicity that $y \in \mathbb{R}^{2N}$ vector, $F= \left[ \begin{array}{cc} F_1 & 0 \\ 0 & F_2 ...
0
votes
0answers
15 views

Is R(A) = ker(A^t)?, where R(A) is the space generated by the columns of A

I'm looking at this deduction of the normal equations that solve the linear least squares problem. It goes like this: R(A) is the space generated by the columns of A $\hat{X}$ is the solution of the ...
2
votes
1answer
19 views

Least squares approximation problem of $t^3$ in a subspace spanned by even degree polynomials.

I am having trouble solving the following question, Let $P_9 ([-1,1])$ be the complex vector space consisting of polynomials $p:[-1,1] \rightarrow\mathbb{C}$ with degree 9 or lower. With the inner ...
0
votes
0answers
16 views

Equivalence of the partial least square regresssion's iterative algorithm and its optimization problem

I am reading The Elements of Statistical Learning. This is a page from the partial least square section: The exercise asks to prove the equivalence between Algorithm 3.3 and Eq. (3.64). Here's my ...
1
vote
0answers
126 views

Can gradient descent solve this problem $\text{argmin}_x \|Ax-[Var(Ax)]^{\frac{1}{2}}-b\|^2$?

How can I find the (approximate) solution to the following problem: $$\text{argmin}_x \|Ax-[Var(Ax)]^{\frac{1}{2}}-b\|^2,$$ where $Var(.)$ denotes the variance? $A$ is matrix and $b$ and $x$ are ...
0
votes
1answer
15 views

Determining parameters for static system

So i have to determine parameters for static system: $y=o1 + o2u$ So, my idea was to pick mesurments for witch $\det[]!=0$ (2 of them as L=1 and R=2) Parameters :$n=1 u=-2 y=9$ and $n=2 u=1 y=-2$ ...
1
vote
0answers
22 views

incremental approach to solve positive least square problem

Is there any incremental (approximate) solution for the following positive least squares problem: $$\min_x \|Ax-b\|^2\qquad \textrm{s.t.}\qquad x_i> 0,~b_1=1,~b_{i>1}=0$$
0
votes
0answers
27 views

Least squares: Calculus to find residual minimizers?

Reading a section on simple regression in "An Introduction to Statistical Learning with Applications in R" I got a question on residual sum of squares minimization. Quoting from the book: [quote] ... ...
1
vote
1answer
45 views

Least-squares in integral form

In the linear least squares problem one minimizes the norm of the residual vector: $$ \chi^2=\|\vec{b}-A\vec{x}\|_2^2=\sum_i^m{|b_i-\sum_j^nA_{ij}x_j|^2} $$ where $A \in \mathbb{R}^{m\times n}, ...
1
vote
0answers
13 views

Is there any way how logistic regression parameters can be preserved under a linear projection?

I have a logistic regression model and OLS estimation of its parameters $\hat{w}$: $$y=\sigma(x^Tw + b)$$ Now, I would like to reduce the dimensionality of the space in which $x \in X$ lies by some ...
0
votes
1answer
27 views

X values for least squares approximation

If I have a yearly quantity (eg. 2000 - 45, 2001-67, 2002 - 38.....2010 - 38) and I need to find the least squares line for this relationship, what should I use for the X values? Should I use the ...
0
votes
1answer
33 views

Least Square fit for signal data (360 points)

I would like to analyze data to get the maximum value out of 360 points. I used least square fitting because I get the data from signal strengths. I want to remove any outliers I get from my data ...
3
votes
0answers
83 views

Tikhonov regularization vs truncated SVD

To find $\mathbf{x}$ such that $$A\mathbf{x}=\mathbf{b}$$ we can use least squares when the problem is not well posed. Further, we can use Tikhonov regularization when $A$ is ill-conditioned. In ...