Questions tagged [weighted-least-squares]

This tag is for questions relating to weighted least squares, a generalization of ordinary least squares and linear regression in which the errors covariance matrix is allowed to be different from an identity matrix.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

Weighted least squares as a weighted mean

Imagine I have my datas $\left(\phi_i\right)$ on a 3D regular grid $\left(x_i,y_i,z_i\right)$ [$n$ values with $8 \leq n \leq 40$ in general] and I want to know $\phi_{v}$ at a virtual point $\left(...
user avatar
  • 6,813
0 votes
0 answers
9 views

MSE of WLS estimator with biased measurements

(also posted on CV, but I will try here too) I am trying to find out if what I am looking at is a known problem. I am considering the case of weighted least squares, and I am trying to find the ...
user avatar
  • 328
1 vote
1 answer
105 views

Asymptotic Normality of Weighted LSE (Theorem 3.17, Jun Shao)

I am trying to understand Jun Shao's proof of the asymptotic normality of weighted LSE in his book Mathematical Statistics. The theorem: Consider the model $X = Z\beta + \varepsilon$ with a full rank $...
user avatar
  • 511
0 votes
1 answer
39 views

Weighted least squares formula

I was studying the weighted least squares algorithm and came across this formula for calculating the weighted result in terms of the original. Here $x_{LS}$ is the solution for $D=I$ I can't figure ...
user avatar
  • 21
3 votes
0 answers
85 views

Best way to remove a local maxima from a piecewise linear function

Let $x_1,...x_K \geq 0$ and $f$ be the piecewise-linear function given by $f(k)=x_k$ for every $1 \leq k \leq K$. Denote by $m$ the number of modes (i.e. local maxima) of $f$. Let's associate with ...
user avatar
  • 69
0 votes
0 answers
23 views

Order of convergence of operators

I was studying the paper "Discrete multi-projection methods for eigen problems of compact integral operator" by G Long, Gnaneshwar Nelakanti, Bijaya L Panigrahi and Mital M Sahani. for my ...
user avatar
0 votes
0 answers
24 views

How to fit a ray into a set of 2D points, having the first point as pivot?

I am trying to improve the line simplification algorithm from Ramer-Douglas-Peucker by adding some kind of line fitting into it. Today, this algorithm does not provide the simplification with ...
user avatar
  • 113
0 votes
0 answers
20 views

WLS/GLS weights when predicted response is zero?

I am working with a nonlinear ODE system where I am attempting to calculate the following: $$\widehat{\theta}_{WLS} = \text{arg}\min_{\theta} \sum_{j=1}^N w_j \left[y_j - f(t_j; \theta)\right]^2$$ ...
user avatar
2 votes
0 answers
39 views

Weighted least squares problem with a equal quadratic constraint

We need to solve the following least square problem $$\min_x (Y-Ax)^TW(Y-Ax)$$ $$s.t. x^TA^TAx=1$$ $$c^Tx=0$$ in a closed form, where $Y \in \mathbb{R}^{n\times 1}$, $A \in \mathbb{R}^{n\times n}$, $W ...
user avatar
0 votes
0 answers
24 views

Optimization problem given weight matrix

Suppose there is a unit norm vector $x \in R^n$, and $y$ is a linear combination of elements of $x$, and let $y = \sum_{i=1}^n w_i^3 x_i$. Given the knowledge of $w_i$, $w = [w_1, ..., w_n]^T$, are we ...
user avatar
  • 151
0 votes
0 answers
23 views

Weighted least squares with sample variance

I am taking a look at some practice problems for the weighted least squares estimator. However I encountered a problem where I am second-guessing what my W matrix should be. I know what the other ...
user avatar
  • 473
0 votes
0 answers
52 views

combining two fit results

I have got two different kind of data points $(x_i, y_i)$. Because one kind are the result of several measurements of the same object I know their statistical error $\Delta x_i, \Delta y_i $ for $x_i$ ...
user avatar
0 votes
1 answer
70 views

How does the bias weight $w_0$ get computed during ridge regression?

I am given a full-rank feature matrix $\mathbf{X}$ to which I am supposed to provide a closed form solution for the weights $\hat{\mathbf{w}}_{ridge}$ of a ridge-regression optimization problem. The ...
user avatar
  • 63
0 votes
0 answers
10 views

How to find out weighted frequency

I am doing a school project on a dataset. I can not understand one of the specific column of the dataset. Data is as follows. This is an instance of whole dataset Question: Define your gender? ...
user avatar
  • 179
1 vote
0 answers
41 views

Using parameter estimation for training a neural network

Assume that we have 4 layers in a neural network. $$z_1 = L_1(x, W_1)$$ $$z_2 = L_2(z_1, W_2)$$ $$z_3 = L_3(z_2, W_3)$$ $$y = L_1(z_3, W_4)$$ Where $x$ is the vector input, $y$ is the vector output ...
user avatar
  • 2,497
1 vote
1 answer
92 views

Writing double summation in matrix form

I have the following weighted residuals sum of squares (WRSS) $$ \sum_{t=1}^n \sum_{i=1}^n \left(y_i - \boldsymbol{x}_i^T \hat{\boldsymbol{\beta}_t} \right)^2 (z_t - z_i) $$ where $y_i \in \mathbb{R}^{...
user avatar
2 votes
1 answer
104 views

Proof of Weighted Least Square Solution

When calculating Weighted Least Square Solution, after taking the derivative, we will have the following equation: $X^\top WX\beta=X^\top Wy$ where $X_{n\times m}$ is the data matrix, with $n\geq m$ ...
user avatar
2 votes
0 answers
29 views

Obtaining all solutions of a linear equation by weighted generalized inverse

Consider a linear equation $$ Ax=b,\quad b\in\operatorname{col}(A). $$ The vector $b$ lies in the column space of $A$ so the solution of this linear equation exists. It is known that any solution has ...
user avatar
0 votes
1 answer
114 views

Why is there more weight on smaller y values in transformed linear regression as compared to least squares regression for exponential models?

I was doing regression for an exponential model that follows the general formula $$ y=Ae^{Bx} $$ And found that using linear regression for the linearized data would model larger values of y poorly ...
user avatar
5 votes
2 answers
178 views

Taylor approximation is not optimal

My professor gave a lecture on an orthogonal polynomial based approximation and its advantage over the Taylor series expansion. And his statement was ``in weighted $L_2$ space, Taylor series expansion ...
user avatar
1 vote
0 answers
20 views

Norm of regularized least square estimator

Let $\theta_*$ be the d-dimensional hidden true parameter, $y_t = x_t^\top \theta_* + \eta_t$ where $\eta_t$ is a standard gaussian noise. It is well known that regularized least square estimator is $\...
user avatar
1 vote
0 answers
39 views

Iteratively reweighted least squares for LASSO problem

I'm trying to solve the following (here simplified) problem (here 1D, and $x>0$): $$ \arg \min_{x} \frac{1}{2} \left\| A x - b \right\|_{2}^{2} + \frac{1}{2} \left\| x \right\|_{1}$$ I need to ...
user avatar
  • 11
1 vote
1 answer
740 views

Variance-Covariance matrix for Weighted Least Squares

For ordinary least squares (OLS), the solution to the system $X\beta = y$ is $\hat{\beta} = (X^T X)^{-1} X^T y$ and the variance on the solution parameters is $Var(\hat{\beta}) = \sigma^2 (X^T X)^{-1}$...
user avatar
  • 53
0 votes
1 answer
62 views

Using linear least-squares to solve this system

I have $\frac{n(n-1)}{2}$ equations of the form: $$2(x_j-x_i)X+2(y_j-y_i)Y+2(z_j-z_i)Z+2v^2\tau(t_i-t_j)=2v^2(t_i^2-t_j^2)+(x_j^2+y_j^2+z_j^2)-(x_i^2+y_i^2+z_i^2)$$ With $n\geq 4$, and where ...
user avatar
  • 1,309
2 votes
1 answer
88 views

Finding point on a plane closest to a point in $\mathbb{R}^n$ using least squares method?

Suppose S is a 2d plane in $\mathbb{R}^3$ s.t. it is the set of all vectors in $\mathbb{R}^3$ with $ax_1+bx_2=0$ (a,b not equal to 0). Let $b=(x_1,x_2,x_3)^T$ be any vector in $\mathbb{R}^3$. How can ...
user avatar
0 votes
1 answer
49 views

How is iteratively reweighted least squares used for $L^p$ norm linear regression?

The iterative scheme that I see everywhere in this context is $$\theta _{k+1}=\left(X^{\:t}W_k\:X\right)^{-1}\left(X^{\:t}W_k\:Y\right)$$ With the weight $W_k$ being a diagonal matrix of $$w_i=\left(...
user avatar
  • 448
0 votes
0 answers
73 views

Solving a coupled system of linear ODEs one second order, the other first order

i want to find Aerodynamic center (y,z) and its moment(M ac) for vehicle , so i searched a lot , and i found these information . Aerodynamic center position doesn't change wrt angle , drag ,lift ...
user avatar
0 votes
1 answer
110 views

Convexity conditions for an Infimum operation

I came across these 2 seemingly contradictory statements in the book 'Convex Optimization' by Boyd. and So the top image is said to be concave. For the simplest case where $n=1$ the function is the ...
user avatar
  • 257
0 votes
1 answer
503 views

SVD decomposition in Weighted Least Square

Consider the Linear Regression below: c_hat=argmin ‖b-Xc‖ (1) Where The Least Square solution is as follows: c_hat=inv(X'X)X'b (2) It is possible to decompose data matrix as ...
user avatar
  • 1
0 votes
1 answer
35 views

Estimate weight/impact of different variables on time

I am trying to estimate digestion time for food given their nutritional content. So the idea is a formula that takes nutritional content of a food as input and outputs time in hours: ...
user avatar
1 vote
0 answers
63 views

Weighted Average Loss, does my approach make sense?

nevermind, no one replies anyway.
user avatar
1 vote
0 answers
55 views

Please verify whether the calculating process of the WLS estimator and the variance is correct or not.

There are two Heteroscedasticity regression models 1. $$ y_i = \beta x_i + \epsilon_i, \quad i=1, \ldots, n $$ where $\epsilon_i$'s are independent and distributed as $\epsilon_i \sim N(0, \sigma^2 ...
user avatar
  • 49
1 vote
0 answers
17 views

How to calculate weight for least squares if both values error is known?

Usually minimizing $$\chi^2 = \sum_i w_i (y_i - f(a_0, a_1, \dots, a_n, x_i))^2$$ where $a_k$ are parameters is done by taking $w_i = 1/\sigma_i^2$ where $\sigma_i$ is a observation error of $y_i$ ...
user avatar
  • 121
0 votes
1 answer
50 views

Restricted Weighted Linear Regression in R

I have the following issue. I would like to run a linear regression imposing a constraint on the weighted coefficients. Let me construct an example: Consider the following cross-sectional regression ...
user avatar
0 votes
0 answers
28 views

Restricted weighted OLS

I would like to run a restricted weighted OLS regression in R. Let me first state the problem mathematically: $\arg\min_{\beta} \sum_{i=1}^{n} w_{i} \lvert y_{i} - \sum_{k=1}^{m} x_{i,k} \beta_{k}\...
user avatar
1 vote
0 answers
53 views

From non-linear least squares to weighted linear least squares

Given an overdetermined linear system $A \in \mathbb{R}^{m \times n}$, $b \in \mathbb{R}^{m \times 1}$. And a non-linear function $f(x)$. Given a non-linear least squares: $$ e^* = \min_g \left\lVert ...
user avatar
1 vote
0 answers
35 views

Iterative least squares for TV-like regularizer

I am trying to implement a TV-like surface regularization in a least squares solver. The formulation is as follows: $E_{surface} = \sum |dA_\theta| $ Background $dA_{\theta} = \frac{\theta_1}{f_x ...
user avatar
  • 11
0 votes
1 answer
214 views

fractional curve fitting of the function $y=a+bx^{\alpha}$

Assume I have a set of data $(x_{i},y_{i})$, $i=1,...,m$. How can we find the best values of the parameters $a$ and $b$ and $\alpha$ such that the curve $y=a+bx^{\alpha}$ best fits the data. This ...
user avatar
  • 2,487
0 votes
0 answers
203 views

Alternatives to least square, overestimate?

Ok so I have a very simple linear system of equations with three unknowns and eight equations. Is there anyone that can tell me if there is a method or some way to find a solution which overestimates ...
user avatar
0 votes
1 answer
87 views

Understanding an identity for least squares regression line gradient

In section 2.2 of this paper, Gelman and Park present the following identity for the gradient of the least squares line through a set of 2D points: ...we recall a simple algebraic identity that ...
user avatar
1 vote
0 answers
195 views

Nearest (with respect to weights) symmetric positive semidefinite matrix

I want to compute the nearest symmetric positive semidefinite matrix, similar as Higham did. But here also weights (given by an inverse co-variance matrix) should be taken into account. So the ...
user avatar
0 votes
0 answers
64 views

How to solve conditional least square?

I'm studying least square, so I can calculate optimal solution at overdetermined system. weighted least square But when some conditions are given, I can't calculate optimal solution(sub?) For ...
user avatar
  • 11
2 votes
1 answer
766 views

Matrix notation for weighted sum of squares

While going through page 1 of Lecture 24: Weighted and Generalized Least Squares [PDF], I got the following questions. Weighted sum of squares is defined as below: $$ \sum_{i = 0}^{n}{w_i(Y_i - X_ib)...
user avatar
  • 31
0 votes
0 answers
215 views

Weighted Least Squares - Categorical Data vs. Numerical Data

Over on Stackoverflow, I am trying calculate the Weighted Least Squares (WLS) of a data set in a python library called Numpy as compared to using a library called Statsmodels. However, I noticed ...
user avatar
0 votes
0 answers
78 views

Minimizing a multivariable function with different vector length?

Assume that I want to minimize a function to a measured vector. $$V_{min} = \sum_{n = 0, k = 0}^{N, K}{|T_n - f(t_n, p_n, r_k)|}$$ Where $T_n, t_n, p_n$ has the same vector length $N$, but not $r_k$ ...
user avatar
  • 2,497
4 votes
2 answers
723 views

How can I get the gradient of the normal equation for weighted linear regression?

The normal equation for weighted linear regression looks like this: $$J(\theta) = (X\theta - y)^TW(X\theta - y),$$ where $X\in\Re^{m\times n}$, $\theta\in\Re^{n\times n}$, $y\in\Re^{m\times 1}$, $W\...
user avatar
  • 199
0 votes
2 answers
55 views

Is there a least squares estimator for correlated, non-constant variance errors?

For OLS we have $\hat{\beta} = (X^TX)^{-1}X^Ty$, For non-constant variance we have $\hat{\beta} = (X^TWX)^{-1}X^TWy$, but what if we have, for example $Y = X\beta + \epsilon $ where $\epsilon \sim N(...
user avatar
  • 3
2 votes
1 answer
2k views

Understanding Galerkin method of weighted residuals

I have a puzzlement regarding the Galerkin method of weighted residuals. The following is taken from the book A Finite Element Primer for Beginners, from chapter 1.1. If I have a one dimensional ...
user avatar
  • 157
0 votes
1 answer
846 views

Weighted Least Squares Without Intercept

I am studying the WLS model for $y=\beta x+\epsilon$, where $\beta$ and $x$ are vectors and $\epsilon$ is the error term. This is a multiple regression model without an intercept. How would I find the ...
user avatar
  • 59
1 vote
1 answer
122 views

polynomial least squares derivation: normal equations

Suppose we have the problem $$ \min_{q \in \mathcal{P}_n} \|f - q\|_{L^2(w)}^2 $$ where $\mathcal{P}_n$ is the space of polynomials of degree $n$, $w$ is some weight function (measure with continuous ...
user avatar
  • 2,370