Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Questions about (linear or nonlinear) least-squares, an estimation method used in statistics, signal processing and elsewhere.

0
votes
1answer
29 views

linear model matrix identification with least squares

I need to do a linear model identification using least squared method. My model to identify is a matrix $A$. My linear system is: $[A]_{_{n \times m}} \cdot [x]_{_{m \times 1}} = [y]_{_{n \times 1}}$ ...
1
vote
1answer
18 views

DEMONSTRATION FINITE-SAMPLE PROPERTIES OF LEAST SQUARES $\frac{(N-k)S^2}{\sigma^2}\sim\chi^2[n-K]$

Im a Student of Economics, and I have a concern. In the solution of $\frac{(n-K)S^2}{\sigma^2}\sim\chi^2[n-K]$ How can I show that if the matrix is ​​symmetric and idempotent between $(I-H)=|| (I-...
0
votes
1answer
24 views

Regularized least squares program and best approximation problem

I have a primal least squre problem of $ min_{w\in R^p} \frac{1}{2} || y - Xw ||_2^2 + \sum_{i=1}^{d} h_i (w_i)$, where $w_i$ are partitions of $w$ and, $w_i \in R^{p_i}$, and let $X_i$ be the ...
0
votes
0answers
34 views

Least squares solution to overdetermined $AX=B$ where each matrix is a rotation

I can't seem to find anything that would help me with this particular problem. I have a bunch of measurements of a matrix $A$ and corresponding matrix $B$, which I know are related by a third rotation ...
1
vote
1answer
29 views

How to find equation for $\theta_\lambda$

Suppose we have data $D=\{x_i,y_i\}_{i=1}^n$ where $x=(x_{i,1},x_{i,2},1)^T \in \mathbb{R}^3$. An estimator for these data, $$y=f(x;\theta)=(\theta_1 x_1+\theta_2 x_2+\theta_3) ~~~(\theta \in \mathbb{...
0
votes
0answers
5 views

Design matrix for multivariate euler function

I want to use the least squares adjustment to get the parameter $c$ and $a$ of the following formula: $$f(a,c) = c \cdot e^{-a^2 \cdot r^2}$$ For the design matrix, I used the taylor series: $$f(x) ...
0
votes
1answer
31 views

What is the estimated value of $\theta_\lambda$?

Suppose we have data $D$ for $(x,y)$ where $D =\{(x_i,y_i)\} = \{(1,1),(a,b)\}$ and we have a estimator $$y=f(x;\theta)=(x;1)\theta=\theta_1x_1+\theta_2$$ for data. I want to minimize $\theta$, so I ...
0
votes
1answer
36 views

Periodic B-spline least squares fitting

I have a set of $M$ data points $(x_j,y_j), 1 \le j \le M$ defined on an interval $[a,b]$ and I want to fit a periodic B-spline of order $k$ to these data. So my model will be: $$ f(x) = \sum_{i=1}^N ...
0
votes
0answers
7 views

How do we solve design system matrix when there is no line that passes through all points

When solving for Ordinary Least Squares regression line I was taught that you solve the system of equations $$ y = X\beta \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;(1)$$ where $X$ is the design matrix. The way ...
0
votes
0answers
10 views

econometrics: weighting frequency over time

I want to measure a model like: Score = arunning_sum_of_all_exercises + brunning_sum_of_identical_exercise + c*exercise_density_value_in_period_x --other variables -- Lets say I want to test ...
0
votes
3answers
64 views

Constrained parameters in least square curve fitting

I have some data points that need to be fit to the curve defined by $$y(x)=\frac{k}{(x+a)^2} - b$$ I have considered that it can be done by the least squares method. However, the analytical solution ...
1
vote
1answer
62 views

How to compute $\frac{\partial}{\partial \theta}L(\theta,\lambda)$?

I'm new to machine learning and currently I'm working on simple linear regression model. But I'm in trouble with computing the squared error. I have $$y=f(x;\theta)=(x,1)\theta=\theta_1 x+ \theta_2$$ ...
1
vote
0answers
26 views

Formulating mathematical model that minimizes the largest deviation between the data and the line

Can someone help me and explain all the steps of solving this problem which is in the field of Model Fitting and Least-Squares Fit. Thank you. PROBLEM: For each of the following data sets, formulate ...
2
votes
0answers
38 views

Tricks for speeding up Conjugate Gradient on Normal Equations?

I have been using Krylov subspace methods for a long time in my research. I know from general theory of Krylov subspace methods that for solving a problem: $$\bf Ax = b$$ if $\bf A$ is square and ...
0
votes
0answers
18 views

Anchoring matrices in linear equation system for uniqueness

In this book "Articulated Motion and Deformable Objects: 7th International Conference" (1) they describe how to get a unique (non-zero) solution for this homogeneous system of linear equations $MR = [....
1
vote
2answers
54 views

Least Squares with equality and inequality constraints

Could someone kindly suggest a method of solving the following constrained (equality and inequality) system of equations in the least squares fashion? $$\min_x\frac 12\|Ax-b\|_2^2$$ such that $$\...
0
votes
0answers
12 views

modifying gumbel function to suit data

I have a set of data (x and y), which is right-skewed. I am now looking for a function that fits the data best (least squares). So far I was thinking about a curve that has a form of a gumbel or log ...
0
votes
0answers
27 views

Find x which minimizes $||B - AX||_F$

I am finding a proper linear algebraic approach to find X that minimizes $||B-AX||_F$ where B satisfies a system of equation TB=H. I have given A, T, and H as matrix from.
4
votes
2answers
116 views

How to solve this inequality-constrained least-squares problem?

I need to solve the following inequality-constrained least-squares problem in vector $x$ $$ \min_{Ax \geq 0} \frac{1}{2} \|Ax-b\|_2^2$$ where matrix $A$ and vector $b$ are given. I am totally ...
-2
votes
1answer
18 views

I can't find the product moment coefficient of correlation? [closed]

These are the equations of least square regression lines: $ Y = 20.8 - 0.219 X $ ($Y$ on $ X$) $X = 16.2 - 0.785 Y$ ($X$ on $Y$) Find the coefficient of correlation $r$.
0
votes
2answers
37 views

Least Square Linear Regression Help

I am given 10 points in the R^3 and my job is to see if the points fit a circle, and if so what is the radius and center. I was not quite sure how to start. Originally, I thought the couldn't because ...
0
votes
1answer
44 views

Why do Least Squares Fitting and Propagation of Uncertainty Derivations Rely on Normal Distribution

In learning more about the Normal distribution, I came upon the following sentence in the first section of the Normal Distribution article on Wikipedia Moreover, many results and methods (such as ...
1
vote
0answers
21 views

MATLAB - Least Squares Fitting for Log - Log Data to find p value.

This is part of the code for a random-walk simulation. To test the code, I'm using steps=[30]; there will be more values, but I want to run it for 1 trial to decrease code processing. ...
0
votes
0answers
22 views

Estimating rate of decay of residual norms in gradient descent

I am using gradient descent to solve the linear system $Ax=b$, where matrix $A$ is symmetric and positive definite. More precisely, I am attempting to solve the following quadratic program $$\text{...
0
votes
1answer
19 views

Normal Equation Derivation Step Help

I was interested in seeing how the normal equations for least squares for linear regression are derived, and found this page: https://eli.thegreenplace.net/2014/derivation-of-the-normal-equation-for-...
0
votes
1answer
46 views

Showing 2 matrices span the same subspace?

Lets say we have a $m \times n$ matrix $A$ and a $m \times n$ matrix $B$. $A$ and $B$ span the same subspace if and only if there is a $n \times n$ matrix $C$ such that $B$ = $AC$. Show that $A$ and ...
0
votes
1answer
51 views

Intuitive explanation of the normal equations for least squares problems

In the least squares method, what does $A^T A$ indicate and, similarly, the product $A ^T b$? That is, why do we multiply both sides of the equation $Ax = b$ by $A^T$? What does it tell us? I know ...
0
votes
0answers
36 views

Computing Maximum Likelihood and Least Squares Estimators for Parameter Estimation of Gaussian Model

I am having trouble with the following question, particularly the first part. Doesn't least squares require that errors be the same across the RV drawn from a distribution. However the variance ...
0
votes
0answers
8 views

How to find basis of functions for best representation of function described in several different truncated ON systems?

I have two ON bases where I to the best $L_2$ precision possible represent the same function. Let us say that these have $n_1,n_2$ basis functions respectively for which we know the coefficient ...
3
votes
1answer
47 views

Localization based on Distance Measurements via Least Squares

Estimating a vector $x \in \mathbb{R}^2$ knowing its distance to four beacons $v_1, \dots, v_4\in \mathbb{R}^2$ via least-squares means finding the least-squares solution to $A x = y$, where $y\in\...
0
votes
1answer
22 views

Matrix equality (related to Tikhonov regularization)

I am trying to prove this equality between matrices: $(A^T A + \mu I_n)A^T=A^T(AA^T+\mu I_m)$ where $A \in \mathbb{R}^{m\times n}, \mu \in \mathbb{R}, \mu > 0$. I was given a hint that I should use ...
2
votes
1answer
31 views

The steps to get $\beta$ in Least Square Estimation, why $x_i$ was removed?

Please see the following steps Get $\beta$ in Least Square Estimation:
0
votes
1answer
27 views

Extending the least square estimation from the vector to a matrix

Suppose each row of the matrix $X$ of $m\times n\,\,(m>n)$ dimension refers to the feature vector of data, and the vector $y$ of $m\times 1$ be the value of each data. As we all know, we can use ...
0
votes
0answers
25 views

Question about convexity of least-squares problem and pseudoinverse

In the rank-deficient case, for A $\in \mathbb{C}^{m\times n}$, the solution of the least-squares problem, with b $\in \mathbb{C}^m $, $$min_{x \in \mathbb{C}}||Ax-b||_2$$ is not unique. (i) Prove ...
1
vote
0answers
22 views

How to choose a proper $\lambda$ for LASSO or BPDN problem?

When I deal with the compressed sensing problem, which can be written as: $min |x|_0 \;\; s.t. \|Ax-y\|^2\le\epsilon$. And there are some way to solve the problem, such as relaxing it as $min \;\...
3
votes
2answers
204 views

Solve matrix equation $XAX^*=B$ for $X$ in least squares sense

Problem How can the following matrix equation be solved $\underset{\mathbf{X}}{\mathrm{argmin}} \left\Vert \mathbf{X}\mathbf{A}\mathbf{X}^* - \mathbf{B} \right\Vert_F$ where $\mathbf{X} \in \...
1
vote
0answers
23 views

From non-linear least squares to weighted linear least squares

Given an overdetermined linear system $A \in \mathbb{R}^{m \times n}$, $b \in \mathbb{R}^{m \times 1}$. And a non-linear function $f(x)$. Given a non-linear least squares: $$ e^* = \min_g \left\lVert ...
1
vote
0answers
24 views

Iterative least squares for TV-like regularizer

I am trying to implement a TV-like surface regularization in a least squares solver. The formulation is as follows: $E_{surface} = \sum |dA_\theta| $ Background $dA_{\theta} = \frac{\theta_1}{f_x ...
1
vote
3answers
40 views

Finding Points to Minimize Distance Between Lines

(Distance between lines) The points $P = (x,x,x)$ and $Q = (y,3y,-1)$ are on two lines in space that don't meet. Choose x and y to minimize the squared distance $||P - Q||^2$. What confuses me here ...
0
votes
1answer
17 views

Calculating the regression coefficient of a time-based series

If i have a sample (lets say house price in millions over time) where x=1,2,...,14 samples y-values are shown in the image below, with one sample estimated per month. How do i calculate the estimated ...
0
votes
0answers
15 views

Can Kabsch's algorithm also provide the covariance of its solution?

Say you have a collection of data points $p_i \in \Bbb R^3$, and a collection of corresponding reference points $r_i \in \Bbb R^3$. Kabsch's algorithm, which relies on SVD decomposition, provides an ...
0
votes
2answers
28 views

Average of a set of values using least squares formula

To get the equation of a line $y = ax+b$ passing through a set of $n$ points $(x_i, y_i)$ using least squares formula, we have to solve the following system of linear equations to determine the ...
1
vote
1answer
40 views

How to minimize the $0$-“norm” with quadratic constraint?

I have a vector $$y = Ax + n$$ where vectors $x, y, n$ are $25 \times 1$, matrix $A$ is $25 \times 25$ and near-orthogonal (actually, it's a part of the DFT matrix). Also, $x$ is sparse and has only ...
1
vote
0answers
17 views

How to solve this weibull distribution equation using least square?

$$f_y=e^{-[(A_y+\delta)/\gamma]^\delta}$$ I'm not sure how to I derive the least square method to solve the unknown parameter for delta and gamma. Any help would be appreciated. What I did tried to ...
1
vote
1answer
94 views

On monotonic quadratic least squares

Quadratic least squares can be used to fit a quadratic curve to $3$ or more points, such that the resulting curve is the quadratic curve that has the least squared distance of the data points to the ...
0
votes
0answers
16 views

How to understand that the solution to least squares problem transformed with Box-Cox Transformation, is a generalized mean with $h(x)=x^\lambda$?

The least squares problem $\min_a \sum_i^n (x_i-a)^2$ is sometimes solved using transformed variables, that is, solving $\min_a \sum_i^n [h(x_i)-h(a)]^2$. The solution to this latter problem is $a=h^{-...
0
votes
1answer
53 views

Solving weighted least squares with non-negative constraints

I have the optimization problem $$ \begin{align} \min_{\mathbf{P} \geq 0} \|\mathbf{A\odot(X-PQ^\top)}\|^2 + \frac{\|\mathbf{P}\|^2}{2} \end{align} $$ $\odot$ is the Hadamard product, $\mathbf{A,X,P,...
0
votes
0answers
23 views

Wassertein and symmetry

So here's a scenario: I have points $(\mu_1^j,\mu_2^j)$ and I associated them the following distribution $$\rho_j=1/2\delta_{\mu_1^j}+1/2\delta_{\mu_2^j}$$ These have symmetry (exchanging $\mu_1^j$ ...
1
vote
1answer
29 views

Using the least squares method for problems with two independent variables

This may be quite a specific question and I apologise however I have struggled to find any information regarding a method. I have 6 given $P_i$ values and 6 given $E_i$ and $F_i$ values and I want to ...
0
votes
0answers
44 views

In this constrained minimization problem, should the Lagrange multipliers be positive?

Consider the following (real, block ?) matrix $Z_{n\times k+1}=[1_{n\times 1},X_{n\times k}]$. Note how $z\equiv v^TZZ^Tv$ can be written as: $v^T11^Tv+v^TXX^Tv=v^TJ_nv+v^TWv$, where $J_n$ is a unit ...