Questions tagged [weighted-least-squares]

This tag is for questions relating to weighted least squares, a generalization of ordinary least squares and linear regression in which the errors covariance matrix is allowed to be different from an identity matrix.

Filter by
Sorted by
Tagged with
0 votes
0 answers
29 views

Time Weighted Decay

I've been tasked to calculate/forecast the weighted exposure of a financial product. I work with bunker prices and we have access to bunker future prices everyday. They look similar to this in an ...
zacchhh's user avatar
2 votes
0 answers
54 views

Weighted Least Square with infinite weights

I am considering a weighted least square problem with data $X \in \mathbb{R}^{n \times p}$, (diagonal) weight matrix $W \in \mathbb{R}^{n \times n}$ and responses $y \in \mathbb{R}^n$, i.e. finding $$\...
Fabi's user avatar
  • 21
0 votes
0 answers
32 views

Linear regression for data points with given deviations?

I have a problem I fail to research properly, so I hope you may at least push me in the right direction (or maybe even provide me an answer right away?). I know how linear regression works, that it ...
AlpaY's user avatar
  • 111
0 votes
0 answers
20 views

Derivation of Feasible Generalized Least Squares Process (Wooldridge 2013)

I'm using Wooldridge's Introductory Econometrics (2013) and having some trouble understanding his derivation of the feasible generalized least squares (FGLS) method. He writes, assume that the ...
user601846's user avatar
0 votes
0 answers
33 views

Weighted Least squares with Multiple Unknowns and Iterations

I am currently working on a problem involving the minimization of the $\chi^2$ deviation between a model matrix ($C_{model}$) and a measured matrix ($C_{measured}$). by finding the best fit parameters ...
Accelerator's user avatar
2 votes
0 answers
121 views

Noise Covariance Estimation for Linear Regression (Seemingly Unrelated Regressions)

Considering following linear model \begin{equation} y_t = X_t f_t + \varepsilon_t, \qquad t=1,\cdots, T \end{equation} where $y_t\in\Re^{300\times 1}$ and $X_t\in\Re^{300\times 60}$ are two given ...
Stephen Ge's user avatar
0 votes
0 answers
54 views

MSE of WLS estimator with biased measurements

(also posted on CV, but I will try here too) I am trying to find out if what I am looking at is a known problem. I am considering the case of weighted least squares, and I am trying to find the ...
smallStackBigFlow's user avatar
1 vote
1 answer
159 views

Asymptotic Normality of Weighted LSE (Theorem 3.17, Jun Shao)

I am trying to understand Jun Shao's proof of the asymptotic normality of weighted LSE in his book Mathematical Statistics. The theorem: Consider the model $X = Z\beta + \varepsilon$ with a full rank $...
mlcv2022's user avatar
  • 663
0 votes
1 answer
149 views

Weighted least squares formula

I was studying the weighted least squares algorithm and came across this formula for calculating the weighted result in terms of the original. Here $x_{LS}$ is the solution for $D=I$ I can't figure ...
Arthur's user avatar
  • 23
3 votes
0 answers
114 views

Best way to remove a local maxima from a piecewise linear function

Let $x_1,...x_K \geq 0$ and $f$ be the piecewise-linear function given by $f(k)=x_k$ for every $1 \leq k \leq K$. Denote by $m$ the number of modes (i.e. local maxima) of $f$. Let's associate with ...
Skywear's user avatar
  • 184
2 votes
0 answers
83 views

Weighted least squares problem with a equal quadratic constraint

We need to solve the following least square problem $$\min_x (Y-Ax)^TW(Y-Ax)$$ $$s.t. x^TA^TAx=1$$ $$c^Tx=0$$ in a closed form, where $Y \in \mathbb{R}^{n\times 1}$, $A \in \mathbb{R}^{n\times n}$, $W ...
user1034188's user avatar
0 votes
0 answers
57 views

combining two fit results

I have got two different kind of data points $(x_i, y_i)$. Because one kind are the result of several measurements of the same object I know their statistical error $\Delta x_i, \Delta y_i $ for $x_i$ ...
LeifSec's user avatar
1 vote
1 answer
308 views

How does the bias weight $w_0$ get computed during ridge regression?

I am given a full-rank feature matrix $\mathbf{X}$ to which I am supposed to provide a closed form solution for the weights $\hat{\mathbf{w}}_{ridge}$ of a ridge-regression optimization problem. The ...
Nero's user avatar
  • 73
1 vote
0 answers
48 views

Using parameter estimation for training a neural network

Assume that we have 4 layers in a neural network. $$z_1 = L_1(x, W_1)$$ $$z_2 = L_2(z_1, W_2)$$ $$z_3 = L_3(z_2, W_3)$$ $$y = L_1(z_3, W_4)$$ Where $x$ is the vector input, $y$ is the vector output ...
euraad's user avatar
  • 2,912
1 vote
1 answer
257 views

Writing double summation in matrix form

I have the following weighted residuals sum of squares (WRSS) $$ \sum_{t=1}^n \sum_{i=1}^n \left(y_i - \boldsymbol{x}_i^T \hat{\boldsymbol{\beta}_t} \right)^2 (z_t - z_i) $$ where $y_i \in \mathbb{R}^{...
MOHAMMED's user avatar
  • 169
2 votes
1 answer
449 views

Proof of Weighted Least Square Solution

When calculating Weighted Least Square Solution, after taking the derivative, we will have the following equation: $X^\top WX\beta=X^\top Wy$ where $X_{n\times m}$ is the data matrix, with $n\geq m$ ...
Leblanc's user avatar
  • 21
2 votes
0 answers
32 views

Obtaining all solutions of a linear equation by weighted generalized inverse

Consider a linear equation $$ Ax=b,\quad b\in\operatorname{col}(A). $$ The vector $b$ lies in the column space of $A$ so the solution of this linear equation exists. It is known that any solution has ...
Alexander Zhang's user avatar
0 votes
1 answer
185 views

Why is there more weight on smaller y values in transformed linear regression as compared to least squares regression for exponential models?

I was doing regression for an exponential model that follows the general formula $$ y=Ae^{Bx} $$ And found that using linear regression for the linearized data would model larger values of y poorly ...
RICEL123's user avatar
5 votes
2 answers
342 views

Taylor approximation is not optimal

My professor gave a lecture on an orthogonal polynomial based approximation and its advantage over the Taylor series expansion. And his statement was ``in weighted $L_2$ space, Taylor series expansion ...
Delayed signal's user avatar
1 vote
0 answers
29 views

Norm of regularized least square estimator

Let $\theta_*$ be the d-dimensional hidden true parameter, $y_t = x_t^\top \theta_* + \eta_t$ where $\eta_t$ is a standard gaussian noise. It is well known that regularized least square estimator is $\...
user2998690's user avatar
1 vote
0 answers
62 views

Iteratively reweighted least squares for LASSO problem

I'm trying to solve the following (here simplified) problem (here 1D, and $x>0$): $$ \arg \min_{x} \frac{1}{2} \left\| A x - b \right\|_{2}^{2} + \frac{1}{2} \left\| x \right\|_{1}$$ I need to ...
T.Etti's user avatar
  • 11
1 vote
1 answer
3k views

Variance-Covariance matrix for Weighted Least Squares

For ordinary least squares (OLS), the solution to the system $X\beta = y$ is $\hat{\beta} = (X^T X)^{-1} X^T y$ and the variance on the solution parameters is $Var(\hat{\beta}) = \sigma^2 (X^T X)^{-1}$...
Sketos's user avatar
  • 53
0 votes
1 answer
70 views

Using linear least-squares to solve this system

I have $\frac{n(n-1)}{2}$ equations of the form: $$2(x_j-x_i)X+2(y_j-y_i)Y+2(z_j-z_i)Z+2v^2\tau(t_i-t_j)=2v^2(t_i^2-t_j^2)+(x_j^2+y_j^2+z_j^2)-(x_i^2+y_i^2+z_i^2)$$ With $n\geq 4$, and where ...
10GeV's user avatar
  • 1,331
2 votes
1 answer
127 views

Finding point on a plane closest to a point in $\mathbb{R}^n$ using least squares method?

Suppose S is a 2d plane in $\mathbb{R}^3$ s.t. it is the set of all vectors in $\mathbb{R}^3$ with $ax_1+bx_2=0$ (a,b not equal to 0). Let $b=(x_1,x_2,x_3)^T$ be any vector in $\mathbb{R}^3$. How can ...
whatDoDo's user avatar
0 votes
1 answer
104 views

How is iteratively reweighted least squares used for $L^p$ norm linear regression?

The iterative scheme that I see everywhere in this context is $$\theta _{k+1}=\left(X^{\:t}W_k\:X\right)^{-1}\left(X^{\:t}W_k\:Y\right)$$ With the weight $W_k$ being a diagonal matrix of $$w_i=\left(...
Malzahar's user avatar
  • 520
0 votes
0 answers
87 views

Solving a coupled system of linear ODEs one second order, the other first order

i want to find Aerodynamic center (y,z) and its moment(M ac) for vehicle , so i searched a lot , and i found these information . Aerodynamic center position doesn't change wrt angle , drag ,lift ...
Ahmed elmenshawie's user avatar
0 votes
1 answer
276 views

Convexity conditions for an Infimum operation

I came across these 2 seemingly contradictory statements in the book 'Convex Optimization' by Boyd. and So the top image is said to be concave. For the simplest case where $n=1$ the function is the ...
DuttaA's user avatar
  • 283
0 votes
1 answer
884 views

Confusing Equality Between Ordinary and Weighted Least Squares After SVD

Consider the linear regression below: $$\begin{align}\hat{c}&=\arg\min‖b-Xc‖&(1)\end{align}$$ Where the least squares solution is as follows: $$\begin{align}\hat{c}&=(X'X)^{-1}X'b&(2)\...
A.HO.A's user avatar
  • 1
0 votes
1 answer
36 views

Estimate weight/impact of different variables on time

I am trying to estimate digestion time for food given their nutritional content. So the idea is a formula that takes nutritional content of a food as input and outputs time in hours: ...
howtopythonpls's user avatar
1 vote
0 answers
63 views

Weighted Average Loss, does my approach make sense?

nevermind, no one replies anyway.
user10829235's user avatar
1 vote
0 answers
64 views

Please verify whether the calculating process of the WLS estimator and the variance is correct or not.

There are two Heteroscedasticity regression models 1. $$ y_i = \beta x_i + \epsilon_i, \quad i=1, \ldots, n $$ where $\epsilon_i$'s are independent and distributed as $\epsilon_i \sim N(0, \sigma^2 ...
abba's user avatar
  • 49
1 vote
0 answers
18 views

How to calculate weight for least squares if both values error is known?

Usually minimizing $$\chi^2 = \sum_i w_i (y_i - f(a_0, a_1, \dots, a_n, x_i))^2$$ where $a_k$ are parameters is done by taking $w_i = 1/\sigma_i^2$ where $\sigma_i$ is a observation error of $y_i$ ...
Nick's user avatar
  • 121
0 votes
1 answer
64 views

Restricted Weighted Linear Regression in R

I have the following issue. I would like to run a linear regression imposing a constraint on the weighted coefficients. Let me construct an example: Consider the following cross-sectional regression ...
Till Furger's user avatar
0 votes
0 answers
48 views

Restricted weighted OLS

I would like to run a restricted weighted OLS regression in R. Let me first state the problem mathematically: $\arg\min_{\beta} \sum_{i=1}^{n} w_{i} \lvert y_{i} - \sum_{k=1}^{m} x_{i,k} \beta_{k}\...
Till Furger's user avatar
1 vote
0 answers
62 views

From non-linear least squares to weighted linear least squares

Given an overdetermined linear system $A \in \mathbb{R}^{m \times n}$, $b \in \mathbb{R}^{m \times 1}$. And a non-linear function $f(x)$. Given a non-linear least squares: $$ e^* = \min_g \left\lVert ...
Jiro's user avatar
  • 577
1 vote
0 answers
40 views

Iterative least squares for TV-like regularizer

I am trying to implement a TV-like surface regularization in a least squares solver. The formulation is as follows: $E_{surface} = \sum |dA_\theta| $ Background $dA_{\theta} = \frac{\theta_1}{f_x ...
Flo Win's user avatar
  • 11
0 votes
1 answer
446 views

fractional curve fitting of the function $y=a+bx^{\alpha}$

Assume I have a set of data $(x_{i},y_{i})$, $i=1,...,m$. How can we find the best values of the parameters $a$ and $b$ and $\alpha$ such that the curve $y=a+bx^{\alpha}$ best fits the data. This ...
Medo's user avatar
  • 3,043
0 votes
0 answers
256 views

Alternatives to least square, overestimate?

Ok so I have a very simple linear system of equations with three unknowns and eight equations. Is there anyone that can tell me if there is a method or some way to find a solution which overestimates ...
George12's user avatar
0 votes
1 answer
117 views

Understanding an identity for least squares regression line gradient

In section 2.2 of this paper, Gelman and Park present the following identity for the gradient of the least squares line through a set of 2D points: ...we recall a simple algebraic identity that ...
Richard Ambler's user avatar
1 vote
0 answers
236 views

Nearest (with respect to weights) symmetric positive semidefinite matrix

I want to compute the nearest symmetric positive semidefinite matrix, similar as Higham did. But here also weights (given by an inverse co-variance matrix) should be taken into account. So the ...
Matthias Wurm's user avatar
0 votes
0 answers
82 views

How to solve conditional least square?

I'm studying least square, so I can calculate optimal solution at overdetermined system. weighted least square But when some conditions are given, I can't calculate optimal solution(sub?) For ...
FarmBoy's user avatar
  • 11
2 votes
1 answer
1k views

Matrix notation for weighted sum of squares

While going through page 1 of Lecture 24: Weighted and Generalized Least Squares [PDF], I got the following questions. Weighted sum of squares is defined as below: $$ \sum_{i = 0}^{n}{w_i(Y_i - X_ib)...
Sonnu's user avatar
  • 31
0 votes
0 answers
318 views

Weighted Least Squares - Categorical Data vs. Numerical Data

Over on Stackoverflow, I am trying calculate the Weighted Least Squares (WLS) of a data set in a python library called Numpy as compared to using a library called Statsmodels. However, I noticed ...
Code Doggo's user avatar
0 votes
0 answers
83 views

Minimizing a multivariable function with different vector length?

Assume that I want to minimize a function to a measured vector. $$V_{min} = \sum_{n = 0, k = 0}^{N, K}{|T_n - f(t_n, p_n, r_k)|}$$ Where $T_n, t_n, p_n$ has the same vector length $N$, but not $r_k$ ...
euraad's user avatar
  • 2,912
5 votes
2 answers
969 views

How can I get the gradient of the normal equation for weighted linear regression?

The normal equation for weighted linear regression looks like this: $$J(\theta) = (X\theta - y)^TW(X\theta - y),$$ where $X\in\Re^{m\times n}$, $\theta\in\Re^{n\times n}$, $y\in\Re^{m\times 1}$, $W\...
quanty's user avatar
  • 209
0 votes
2 answers
63 views

Is there a least squares estimator for correlated, non-constant variance errors?

For OLS we have $\hat{\beta} = (X^TX)^{-1}X^Ty$, For non-constant variance we have $\hat{\beta} = (X^TWX)^{-1}X^TWy$, but what if we have, for example $Y = X\beta + \epsilon $ where $\epsilon \sim N(...
noname's user avatar
  • 3
4 votes
1 answer
3k views

Understanding Galerkin method of weighted residuals

I have a puzzlement regarding the Galerkin method of weighted residuals. The following is taken from the book A Finite Element Primer for Beginners, from chapter 1.1. If I have a one dimensional ...
Ohm's user avatar
  • 177
0 votes
1 answer
1k views

Weighted Least Squares Without Intercept

I am studying the WLS model for $y=\beta x+\epsilon$, where $\beta$ and $x$ are vectors and $\epsilon$ is the error term. This is a multiple regression model without an intercept. How would I find the ...
Alvin's user avatar
  • 59
1 vote
1 answer
170 views

polynomial least squares derivation: normal equations

Suppose we have the problem $$ \min_{q \in \mathcal{P}_n} \|f - q\|_{L^2(w)}^2 $$ where $\mathcal{P}_n$ is the space of polynomials of degree $n$, $w$ is some weight function (measure with continuous ...
jjjjjj's user avatar
  • 2,631
1 vote
1 answer
265 views

Weighted Least Squares for Parabola Coefficients Estimation

I am in trouble to find where I am making a mistake... I have to estimate the parameters a and b of the curve modeled by: $y = a x^2 + bx$ I have to do that from K measures of the curve, each ...
Artur's user avatar
  • 13