Questions tagged [weighted-least-squares]
This tag is for questions relating to weighted least squares, a generalization of ordinary least squares and linear regression in which the errors covariance matrix is allowed to be different from an identity matrix.
61
questions
0
votes
0
answers
11
views
Weighted least squares as a weighted mean
Imagine I have my datas $\left(\phi_i\right)$ on a 3D regular grid $\left(x_i,y_i,z_i\right)$ [$n$ values with $8 \leq n \leq 40$ in general] and I want to know $\phi_{v}$ at a virtual point $\left(...
- 6,813
0
votes
0
answers
9
views
MSE of WLS estimator with biased measurements
(also posted on CV, but I will try here too)
I am trying to find out if what I am looking at is a known problem.
I am considering the case of weighted least squares, and I am trying to find the ...
- 328
1
vote
1
answer
105
views
Asymptotic Normality of Weighted LSE (Theorem 3.17, Jun Shao)
I am trying to understand Jun Shao's proof of the asymptotic normality of weighted LSE in his book Mathematical Statistics.
The theorem: Consider the model $X = Z\beta + \varepsilon$ with a full rank $...
- 511
0
votes
1
answer
39
views
Weighted least squares formula
I was studying the weighted least squares algorithm and came across this formula for calculating the weighted result in terms of the original.
Here $x_{LS}$ is the solution for $D=I$
I can't figure ...
- 21
3
votes
0
answers
85
views
Best way to remove a local maxima from a piecewise linear function
Let $x_1,...x_K \geq 0$ and $f$ be the piecewise-linear function given by $f(k)=x_k$ for every $1 \leq k \leq K$. Denote by $m$ the number of modes (i.e. local maxima) of $f$.
Let's associate with ...
- 69
0
votes
0
answers
23
views
Order of convergence of operators
I was studying the paper "Discrete multi-projection methods for eigen problems of compact integral operator" by G Long, Gnaneshwar Nelakanti, Bijaya L Panigrahi and Mital M Sahani. for my ...
0
votes
0
answers
24
views
How to fit a ray into a set of 2D points, having the first point as pivot?
I am trying to improve the line simplification algorithm from Ramer-Douglas-Peucker by adding some kind of line fitting into it.
Today, this algorithm does not provide the simplification with ...
- 113
0
votes
0
answers
20
views
WLS/GLS weights when predicted response is zero?
I am working with a nonlinear ODE system where I am attempting to calculate the following:
$$\widehat{\theta}_{WLS} = \text{arg}\min_{\theta} \sum_{j=1}^N w_j \left[y_j - f(t_j; \theta)\right]^2$$ ...
- 108
2
votes
0
answers
39
views
Weighted least squares problem with a equal quadratic constraint
We need to solve the following least square problem
$$\min_x (Y-Ax)^TW(Y-Ax)$$
$$s.t. x^TA^TAx=1$$
$$c^Tx=0$$
in a closed form, where $Y \in \mathbb{R}^{n\times 1}$, $A \in \mathbb{R}^{n\times n}$, $W ...
- 23
0
votes
0
answers
24
views
Optimization problem given weight matrix
Suppose there is a unit norm vector $x \in R^n$, and $y$ is a linear combination of elements of $x$, and let $y = \sum_{i=1}^n w_i^3 x_i$. Given the knowledge of $w_i$, $w = [w_1, ..., w_n]^T$, are we ...
- 151
0
votes
0
answers
23
views
Weighted least squares with sample variance
I am taking a look at some practice problems for the weighted least squares estimator. However I encountered a problem where I am second-guessing what my W matrix should be.
I know what the other ...
- 473
0
votes
0
answers
52
views
combining two fit results
I have got two different kind of data points $(x_i, y_i)$. Because one kind are the result of several measurements of the same object I know their statistical error $\Delta x_i, \Delta y_i $ for $x_i$ ...
- 1
0
votes
1
answer
70
views
How does the bias weight $w_0$ get computed during ridge regression?
I am given a full-rank feature matrix $\mathbf{X}$ to which I am supposed to provide a closed form solution for the weights $\hat{\mathbf{w}}_{ridge}$ of a ridge-regression optimization problem. The ...
- 63
0
votes
0
answers
10
views
How to find out weighted frequency
I am doing a school project on a dataset. I can not understand one of the specific column of the dataset.
Data is as follows. This is an instance of whole dataset
Question: Define your gender?
...
- 179
1
vote
0
answers
41
views
Using parameter estimation for training a neural network
Assume that we have 4 layers in a neural network.
$$z_1 = L_1(x, W_1)$$
$$z_2 = L_2(z_1, W_2)$$
$$z_3 = L_3(z_2, W_3)$$
$$y = L_1(z_3, W_4)$$
Where $x$ is the vector input, $y$ is the vector output ...
- 2,497
1
vote
1
answer
92
views
Writing double summation in matrix form
I have the following weighted residuals sum of squares (WRSS)
$$ \sum_{t=1}^n \sum_{i=1}^n \left(y_i - \boldsymbol{x}_i^T \hat{\boldsymbol{\beta}_t} \right)^2 (z_t - z_i) $$
where $y_i \in \mathbb{R}^{...
- 47
2
votes
1
answer
104
views
Proof of Weighted Least Square Solution
When calculating Weighted Least Square Solution, after taking the derivative, we will have the following equation:
$X^\top WX\beta=X^\top Wy$
where $X_{n\times m}$ is the data matrix, with $n\geq m$ ...
- 21
2
votes
0
answers
29
views
Obtaining all solutions of a linear equation by weighted generalized inverse
Consider a linear equation
$$
Ax=b,\quad b\in\operatorname{col}(A).
$$
The vector $b$ lies in the column space of $A$ so the solution of this linear equation exists.
It is known that any solution has ...
- 363
0
votes
1
answer
114
views
Why is there more weight on smaller y values in transformed linear regression as compared to least squares regression for exponential models?
I was doing regression for an exponential model that follows the general formula
$$ y=Ae^{Bx} $$
And found that using linear regression for the linearized data would model larger values of y poorly ...
- 31
5
votes
2
answers
178
views
Taylor approximation is not optimal
My professor gave a lecture on an orthogonal polynomial based approximation and its advantage over the Taylor series expansion. And his statement was ``in weighted $L_2$ space, Taylor series expansion ...
- 185
1
vote
0
answers
20
views
Norm of regularized least square estimator
Let $\theta_*$ be the d-dimensional hidden true parameter,
$y_t = x_t^\top \theta_* + \eta_t$ where $\eta_t$ is a standard gaussian noise.
It is well known that regularized least square estimator is $\...
- 109
1
vote
0
answers
39
views
Iteratively reweighted least squares for LASSO problem
I'm trying to solve the following (here simplified) problem (here 1D, and $x>0$):
$$ \arg \min_{x} \frac{1}{2} \left\| A x - b \right\|_{2}^{2} + \frac{1}{2} \left\| x \right\|_{1}$$
I need to ...
- 11
1
vote
1
answer
740
views
Variance-Covariance matrix for Weighted Least Squares
For ordinary least squares (OLS), the solution to the system $X\beta = y$ is
$\hat{\beta} = (X^T X)^{-1} X^T y$
and the variance on the solution parameters is
$Var(\hat{\beta}) = \sigma^2 (X^T X)^{-1}$...
- 53
0
votes
1
answer
62
views
Using linear least-squares to solve this system
I have $\frac{n(n-1)}{2}$ equations of the form:
$$2(x_j-x_i)X+2(y_j-y_i)Y+2(z_j-z_i)Z+2v^2\tau(t_i-t_j)=2v^2(t_i^2-t_j^2)+(x_j^2+y_j^2+z_j^2)-(x_i^2+y_i^2+z_i^2)$$
With $n\geq 4$, and where ...
- 1,309
2
votes
1
answer
88
views
Finding point on a plane closest to a point in $\mathbb{R}^n$ using least squares method?
Suppose S is a 2d plane in $\mathbb{R}^3$ s.t. it is the set of all vectors in $\mathbb{R}^3$ with $ax_1+bx_2=0$ (a,b not equal to 0). Let $b=(x_1,x_2,x_3)^T$ be any vector in $\mathbb{R}^3$. How can ...
- 35
0
votes
1
answer
49
views
How is iteratively reweighted least squares used for $L^p$ norm linear regression?
The iterative scheme that I see everywhere in this context is
$$\theta _{k+1}=\left(X^{\:t}W_k\:X\right)^{-1}\left(X^{\:t}W_k\:Y\right)$$
With the weight $W_k$ being a diagonal matrix of $$w_i=\left(...
- 448
0
votes
0
answers
73
views
Solving a coupled system of linear ODEs one second order, the other first order
i want to find Aerodynamic center (y,z) and its moment(M ac) for vehicle , so i searched a lot , and i found these information .
Aerodynamic center position doesn't change wrt angle , drag ,lift ...
0
votes
1
answer
110
views
Convexity conditions for an Infimum operation
I came across these 2 seemingly contradictory statements in the book 'Convex Optimization' by Boyd.
and
So the top image is said to be concave.
For the simplest case where $n=1$ the function is the ...
- 257
0
votes
1
answer
503
views
SVD decomposition in Weighted Least Square
Consider the Linear Regression below:
c_hat=argmin ‖b-Xc‖ (1)
Where The Least Square solution is as follows:
c_hat=inv(X'X)X'b (2)
It is possible to decompose data matrix as
...
- 1
0
votes
1
answer
35
views
Estimate weight/impact of different variables on time
I am trying to estimate digestion time for food given their nutritional content.
So the idea is a formula that takes nutritional content of a food as input and outputs time in hours:
...
- 101
1
vote
0
answers
63
views
1
vote
0
answers
55
views
Please verify whether the calculating process of the WLS estimator and the variance is correct or not.
There are two Heteroscedasticity regression models
1.
$$ y_i = \beta x_i + \epsilon_i, \quad i=1, \ldots, n $$
where $\epsilon_i$'s are independent and distributed as $\epsilon_i \sim N(0, \sigma^2 ...
- 49
1
vote
0
answers
17
views
How to calculate weight for least squares if both values error is known?
Usually minimizing $$\chi^2 = \sum_i w_i (y_i - f(a_0, a_1, \dots, a_n, x_i))^2$$ where $a_k$ are parameters is done by taking $w_i = 1/\sigma_i^2$ where $\sigma_i$ is a observation error of $y_i$ ...
- 121
0
votes
1
answer
50
views
Restricted Weighted Linear Regression in R
I have the following issue. I would like to run a linear regression imposing a constraint on the weighted coefficients. Let me construct an example:
Consider the following cross-sectional regression
...
0
votes
0
answers
28
views
Restricted weighted OLS
I would like to run a restricted weighted OLS regression in R. Let me first state the problem mathematically:
$\arg\min_{\beta} \sum_{i=1}^{n} w_{i} \lvert y_{i} - \sum_{k=1}^{m} x_{i,k} \beta_{k}\...
1
vote
0
answers
53
views
From non-linear least squares to weighted linear least squares
Given an overdetermined linear system $A \in \mathbb{R}^{m \times n}$, $b \in \mathbb{R}^{m \times 1}$. And a non-linear function $f(x)$. Given a non-linear least squares:
$$ e^* = \min_g \left\lVert ...
1
vote
0
answers
35
views
Iterative least squares for TV-like regularizer
I am trying to implement a TV-like surface regularization in a least squares solver. The formulation is as follows:
$E_{surface} = \sum |dA_\theta| $
Background
$dA_{\theta} = \frac{\theta_1}{f_x ...
- 11
0
votes
1
answer
214
views
fractional curve fitting of the function $y=a+bx^{\alpha}$
Assume I have a set of data $(x_{i},y_{i})$, $i=1,...,m$. How can we find the best values of the parameters $a$ and $b$ and $\alpha$ such that the curve $y=a+bx^{\alpha}$ best fits the data.
This ...
- 2,487
0
votes
0
answers
203
views
Alternatives to least square, overestimate?
Ok so I have a very simple linear system of equations with three unknowns and eight equations. Is there anyone that can tell me if there is a method or some way to find a solution which overestimates ...
- 55
0
votes
1
answer
87
views
Understanding an identity for least squares regression line gradient
In section 2.2 of this paper, Gelman and Park present the following identity for the gradient of the least squares line through a set of 2D points:
...we recall a simple algebraic identity that ...
- 1,608
1
vote
0
answers
195
views
Nearest (with respect to weights) symmetric positive semidefinite matrix
I want to compute the nearest symmetric positive semidefinite matrix, similar as Higham did. But here also weights (given by an inverse co-variance matrix) should be taken into account.
So the ...
0
votes
0
answers
64
views
How to solve conditional least square?
I'm studying least square, so I can calculate optimal solution at overdetermined system. weighted least square
But when some conditions are given, I can't calculate optimal solution(sub?)
For ...
- 11
2
votes
1
answer
766
views
Matrix notation for weighted sum of squares
While going through page 1 of Lecture 24: Weighted and Generalized Least Squares [PDF], I got the following questions.
Weighted sum of squares is defined as below:
$$ \sum_{i = 0}^{n}{w_i(Y_i - X_ib)...
- 31
0
votes
0
answers
215
views
Weighted Least Squares - Categorical Data vs. Numerical Data
Over on Stackoverflow, I am trying calculate the Weighted Least Squares (WLS) of a data set in a python library called Numpy as compared to using a library called Statsmodels. However, I noticed ...
- 219
0
votes
0
answers
78
views
Minimizing a multivariable function with different vector length?
Assume that I want to minimize a function to a measured vector.
$$V_{min} = \sum_{n = 0, k = 0}^{N, K}{|T_n - f(t_n, p_n, r_k)|}$$
Where $T_n, t_n, p_n$ has the same vector length $N$, but not $r_k$ ...
- 2,497
4
votes
2
answers
723
views
How can I get the gradient of the normal equation for weighted linear regression?
The normal equation for weighted linear regression looks like this:
$$J(\theta) = (X\theta - y)^TW(X\theta - y),$$ where $X\in\Re^{m\times n}$, $\theta\in\Re^{n\times n}$, $y\in\Re^{m\times 1}$, $W\...
- 199
0
votes
2
answers
55
views
Is there a least squares estimator for correlated, non-constant variance errors?
For OLS we have $\hat{\beta} = (X^TX)^{-1}X^Ty$,
For non-constant variance we have $\hat{\beta} = (X^TWX)^{-1}X^TWy$,
but what if we have, for example $Y = X\beta + \epsilon $ where $\epsilon \sim N(...
- 3
2
votes
1
answer
2k
views
Understanding Galerkin method of weighted residuals
I have a puzzlement regarding the Galerkin method of weighted residuals. The following is taken from the book A Finite Element Primer for Beginners, from chapter 1.1.
If I have a one dimensional ...
- 157
0
votes
1
answer
846
views
Weighted Least Squares Without Intercept
I am studying the WLS model for $y=\beta x+\epsilon$, where $\beta$ and $x$ are vectors and $\epsilon$ is the error term. This is a multiple regression model without an intercept. How would I find the ...
- 59
1
vote
1
answer
122
views
polynomial least squares derivation: normal equations
Suppose we have the problem
$$
\min_{q \in \mathcal{P}_n} \|f - q\|_{L^2(w)}^2
$$
where $\mathcal{P}_n$ is the space of polynomials of degree $n$, $w$ is some weight function (measure with continuous ...
- 2,370