Questions tagged [linear-regression]
For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.
1,320
questions
0
votes
0
answers
22
views
Bishop gradient calculation
In section 3.1.1 of Pattern Recognition and Machine Learning by Christopher Bishop, it is written that
$$\ln p(\mathbf{t} | \mathbf{w}, \beta) = \frac{N}{2} \ln \beta - \frac{N}{2} \ln (2 \pi) - \beta ...
-1
votes
0
answers
12
views
Passing a linear regression equation around 5 data points, making sure the regression equation passes through the fifth point. [closed]
I have a five point data set which was created by taking the natural log of a data set where x = ln(a) and y = ln(b). The transformed data set is x = {1.61, 2.30, 2.71, 3.40, 4.09} and y = {2.05, 1.43,...
- 7
0
votes
0
answers
25
views
Regression coefficients without matrix inversion
We can find coefficients β in multiple linear regression:
β = (XTX)-1XTy
I see a lot of related questions on this forum asking "How do I invert this matrix?". This is followed by an answer ...
- 109
1
vote
0
answers
38
views
Why can Total Least Squares Regression not have a solution?
I was looking at this reference on TLS regression. I understand the math which says that there can be problems where no solution exists. Based on my understanding, the minimum perturbation to the $C$ ...
- 707
0
votes
0
answers
82
views
How to calculate the trace of this matrix?
How do we calculate the trace of $(\mathbb{I}-X(X^\prime X)^{-1}X^\prime)\mathbb{J}_n$? This question stemmed from the below problem I came across:
Suppose we have the linear regression model: $y_i=...
- 403
0
votes
1
answer
19
views
Help in understanding Bayesian linear regression.
I am unable to use Bayesian Regression in the following
question. So far I have performed the following calculations:
We have our data points Y, X that are ...
1
vote
2
answers
31
views
Why does Least Absolute Deviation regression exactly fit n measureemnts for a linear system with n independent variables?
I am applying LAD regression to conduct some research. I have the following questions regarding LAD:
I know LAD exactly fits n measurements for a linear system with n variables. But I cannot easily ...
- 21
0
votes
0
answers
15
views
Prediction intervals for simple linear regression
I want to understand how to construct the test statistic for the case of predictive inference for a simple linear regression model and would be grateful if someone might confirm if my derivation is ...
1
vote
1
answer
34
views
$R^2$ of regression of residuals when adding an uncorrelated regressor
Suppose we linearly regress $Y$ onto $X_1$, obtaining residuals $\epsilon_1,\ldots,\epsilon_n$. Suppose further that $X_2$ is uncorrelated with $X_1$, and we linearly regress $X_2$ onto $X_1$ (with ...
- 9,378
1
vote
1
answer
32
views
Local Linear Fit
What do you call it when you have a set of known $(x,y)$ data points and you estimate a $y$-value for a given $x$-value by performing a linear fit between its two known neighbor $(x,y)$ points?
As an ...
- 238
0
votes
0
answers
84
views
Continuous piecewise linear (CPWL) function fitting a dataset for which each linear piece contains at least 3 points
Imagine a 2D dataset $(x_i,y_i)_{i=1,...,N}$ and a univariate continuous piecewise linear (CPWL) function composed of $K$ linear pieces $f$ such each point $i$ belongs to the segment $s(i) \in \{1,...,...
0
votes
0
answers
18
views
Removing Independent Variable Uncorrelated with All Other Variables in Linear Regression
I've been looking at Wooldrige's Introductory Econometrics and came across the following section related to omitted variables in multiple regression here
The section essentially says that if an ...
0
votes
0
answers
31
views
Using complex phases to do linear regression
The following is non-standard but was interested to see if there is value in following this path.
Consider a linear regression problem without intercept, so simply, $y=ax$ and some data is provided $(...
- 307
1
vote
0
answers
23
views
Applying Block Bootstrap to Simulate Returns and Conduct OLS Regression for Beta Calculation [closed]
I am facing a methodological issue in my Master's thesis and hope someone can provide some guidance. Background: I have a time series of returns for the S&P 500, a variance swap, and a put option ...
- 11
0
votes
0
answers
22
views
How do nonlinear relationships affect casuality determination
Let's assume that I have only one independent variable and one dependent, and I have a great model with minimal error which deals well with predicting. Let's also assume that I do no know the true ...
- 91
0
votes
0
answers
46
views
Appropriateness of one observation per unique combination of dummy variables
I am wondering what conclusions you can draw regarding the coefficients of an OLS model when you only have one observation per combination of unique dummy variables.
I have seen someone else do this ...
0
votes
0
answers
29
views
Proportion of variance in a linear regression model with a covaring predictor
Given a model:
\begin{align}Y_{i}=Z_{i}*\beta * X_{i} + Z_{i}\tag{Eq. 1}&\end{align}
I am interested in a closed formula for the proportion of variance explained by the predictor variable $X$, ...
- 181
1
vote
1
answer
154
views
Can anyone see a way to linearize this function for linear regression?
I have the following function:
$$f(x) = \dfrac{a_1}{(x+b_1)^2+c_1} + \dfrac{a_2}{(x+b_2)^2+c_2}.$$
From multiple measurements of $f$ at known $x$ values I would like find the values of $a_1,a_2,b_1,...
- 23
0
votes
0
answers
23
views
Correlation vs Regression for a simple task
Hello everyone and thank you for taking the time with my issue! I want to apologize in advance if my question would've fit better on stack exchange, but I decided that the question is more related to ...
- 1
3
votes
0
answers
71
views
Prove $Max(L_1,L_2,L_3)\neq L_2$:$L_i=\frac{S_i^2}{n_i}+\frac{(S-S_i)^2}{n-n_i}$;$S=\sum S_i$;$n=\sum n_i$;$\frac{S_i}{n_i}>\frac{S_{i-1}}{n_{i-1}}$
I will state my question first, and after that, I will write how I arrived to it.
You do not really need to see how I arrived to the question, but I just thought it would be rude not to explain that.
...
- 31
1
vote
2
answers
56
views
Weighted Least Squares versus ordinary least squares wiki page
If I have a $(X, Y)$ dataset and want to model $y = f(x, \beta)$. In that case for OLS, I would have
$$e(x_i, \beta) = f(x_i, \beta) - y_i$$
Then obviously I would have
$$SSE = \sum_{i}e(x_i, \beta)^2$...
- 549
0
votes
0
answers
16
views
Use WLS on difference of data to calculate slope, and OLS for intercept
I have a sequence of data observed in the past X days, which I want to assign higher weights to more recent ones so it makes sense to use WLS over OLS.
However, does it makes sense to use WLS to just ...
- 39
0
votes
1
answer
34
views
Linear regression solution comparison
I tried to solve this exercise several times now using different methods but I am not really sure if my solution is correct.
Please help me and tell me what your solution to this problem is, so I can ...
1
vote
0
answers
10
views
Understanding the Equivalence Conditions between Variance of Estimators in Linear Regression Models
Let $Y=\beta X+\epsilon$ be standard linear model, $\hat{\beta}$ and $\tilde{\beta}$ estimators of $\beta$ and $a\in R^{p}$.
We know that:
$$
Var(a'\tilde{\beta}) \le Var(a'\hat{\beta}).
$$
Is the ...
- 333
1
vote
0
answers
29
views
Is there a continuous time analog of linear regression for SDEs?
The ordinary linear regression
$$Y_n = \alpha+\beta X_n +\epsilon_n$$
has a closed form solution for $\beta$
$$\beta = \frac{\operatorname{Cov}(X, Y)}{\sigma_X^2}.$$
Question:
Is there a continuous ...
- 1,161
0
votes
2
answers
122
views
Converting a multivariate linear correlation into a univariate one (predicting real estate prices in NY)
I am building a prediction model of real estate properties in New York based on a few inputs:
Area (size) of the property
Year it was built
Number of bedrooms
Number of parking spots
The data is ...
- 1,947
0
votes
0
answers
15
views
Verifying the Distribution of a Standardized Least Squares Estimator in Simple Linear Regression
I have the following linear regression,
$$
Y_i=\beta_0+\beta_1 x_i+\epsilon_i
$$
where $\epsilon_i^{\prime}$ s are independent $N\left(0, \sigma^2\right)$ random variables. Let $\hat{\beta}_i$ denote ...
- 461
0
votes
1
answer
32
views
Omitted Variable Bias- linear regression
My question relates to determining the direction of bias when the regression coefficient.
The original simple linear regression model gives a coefficient $\beta_1= − 0.002108$
After including an ...
user1318281
1
vote
0
answers
15
views
A corollary of Frisch-Waugh-Lowell Theorem
The formulation is just a special case of FWL. Say we have a partitioned regression, $Y=X_1\beta_1+X_2\beta_2+\epsilon$ but with $X_2$ be $n\times 1$ and $\beta_2$ a constant.
Let $b_1,b_2$ be two OLS ...
- 305
1
vote
1
answer
44
views
Linear models in statistics (Rencher), Theorem 7.9.c
In multiple regression context (under the general assumption we know), $X=(X_1,X_2)$ and $\beta^\prime=(\beta_1^\prime,\beta_2^\prime)$. Let $\hat{\beta_1^*}=(X_1^\prime X_1)^{-1}X_1^\prime y$ be the ...
- 11
0
votes
0
answers
24
views
Linear Regression with gaussian mixture prior
In linear regression, we assume that the output variable is Normally distributed, i.e., $p(y) = N (y | \mathbf{w}^T\mathbf{x}, \sigma^2_y)$. I want to assign a mixture of Gaussian prior to each ...
- 15
1
vote
0
answers
14
views
Repository of linear and nonlinear base functions for linear regression
is there any repository of linear and nonlinear base functions. I would like to compare some of them according to possibility to represent data. I think about a set as following:
Name
Dimension
...
- 11
1
vote
0
answers
27
views
How to use Multiple Regression Analysis to get to linear equation
I am working on a paper about MQ gas sensors and found this other study (https://www.researchgate.net/publication/...
- 11
0
votes
0
answers
7
views
Confidence Interval for Regression Values
From this article, I understand the idea behind the Confidence Interval for the Individual Response.
However I don't understand the part on ...
- 15
0
votes
0
answers
20
views
Express the regularized weight in ridge regression in terms of the linear regression solution .
We would like to minimize the quantity
$E_{in}(\vec{w})=\frac{1}{N}\sum_{i=1}^N(\vec{w}^{T}\vec{x_n}-y_n)^2$
under the constraint
$\vec{w}^T\Gamma^T\Gamma\vec{w}\leq C$ where $\Gamma$ is a matrix, $C$ ...
- 11
1
vote
0
answers
59
views
Is the expectation of the error of any projection of $\mathbb{E}[Y\mid X]$ onto subspace zero?
If we consider the following linear predictor of $Y$ based on $X$:
$$
Y_{\mathbf{b}}=\boldsymbol{\Sigma}_{Y, \mathbf{X}} \boldsymbol{\Sigma}_{\mathbf{X}}^{-1}\left(\mathbf{X}-\boldsymbol{\mu}_{\mathbf{...
- 617
0
votes
0
answers
11
views
Finding parameters for function approximation
I am working on some project in Matlab, where I defined some function $R(x, h, H, L, N)$ and I want to find such $h, H, L, N$ such that $R(x, h, H, L, N)$ is approximated by some sine wave in other ...
- 171
0
votes
0
answers
21
views
closed form expression for training error in ridge regression
I'm reading the paper a random matrix approach to neural network but i'm stucked at page 5.
They start from
$\Sigma \in \mathbb{R}^{n\times T}$ where $T$ is the number of data points and $n$ is the ...
- 155
0
votes
1
answer
169
views
Can a (bounded) linear least-squares problem include a scale factor in its solution?
I have a system of equations.
$$
\small
\begin{aligned}
(1 - p_0)u_0 + (q_0 - 1)v_0 + u_1 - v_1 = r_0 - c t_0 \\\\
(1 - p_1)u_1 + (q_1 - 1)v_1 + u_0 - v_0 = r_1 - c t_1 \\\\
(1 - p_2)u_2 + (q_2 - 1)...
- 103
0
votes
0
answers
22
views
Derivation of normal equation for linear regression parameters
I'm going through a derivation of the normal equation for the parameter vector $\beta$ of the linear regression model. Given a model $y = X\beta + \epsilon$, where $y$ is the vector of dependent ...
- 183
0
votes
0
answers
60
views
Is conditional expectation of the error of best linear predictor given $X$ is $0$ (Is it true that $y = a^*+b^*x + \eta$, where $E[\eta|x]=0$)?
For simplicity, assume we are working with simple regression where the predictor $x\in\mathbb{R}$.
First write $y=E[y \mid x]+u$, where the variance of $u$ is a constant, and $E[u|x]=0$. I understand $...
- 617
0
votes
0
answers
14
views
Results of linear regression with just statistically significant features vs. whole dataset
If I have a dataset, in this case the diabetes toy dataset, and am running a linear regression model, could someone explain what I should expect in terms of performance if I were to conduct the ...
0
votes
0
answers
26
views
PDF Being Invariant Under Orthogonal Transformations
Regarding linear regression, in the (informal) solution manual of T. Hastie's "The Elements of Statistical Learning" book, there is a paragraph (p. 16 in the manual) that goes:
"... ...
0
votes
1
answer
47
views
Population OLS coefficient in simple regression?
The population OLS coefficient for some $X_i \in \mathbb{R}^d, Y \in \mathbb{R}$ for the model $Y = \beta’X + e$ is defined as
$$
\beta =\mathbb{E}[X_iX_i']^{-1}\mathbb{E}[X_iY_i]
$$
and if $X$ is a ...
- 196
2
votes
2
answers
203
views
Is there a rigorous probability theoretic formulation of linear regression
Let $(\Omega,\Sigma,P)$ be some probability space and let $Y: \Omega \to \mathbb{R} $ be a random variable. Let $x, \beta \in \mathbb{R}^n$. In a linear model we assume something like this: $E(Y|x)=\...
- 1,560
0
votes
0
answers
18
views
Creating a regression model of scaling data from five points for ATAR
I am looking to calculate the scaled results for the ATAR (Australian Tertiary Admission Rank) subjects that someone inputs from the recently released 2023 data. For example if someone got a 59.00 in ...
- 3
0
votes
0
answers
47
views
Simpson's Paradox for $R^2$
I've been reading about Simpson's Paradox and I understand how if you fit a regression on a per-group basis versus on the entire set, the coefficients in each group can be drastically different from ...
- 63
0
votes
0
answers
21
views
Convergence of coefficients in multivariate regression
In this thread, the convergence of coefficient for univariate dependent variable is proven. I wonder, assuming the same setup, how can the convergence be extended to multivariate as:
$$Y=XW+\epsilon$$
...
- 253
0
votes
0
answers
61
views
Proving product of two projection matrices is commutative
I’ve been trying to understand the details of orthogonal projections within the context of linear regression models, and I’ve come across the following scenario:
Given a linear regression model $Y=\...
- 777
4
votes
1
answer
130
views
Sqrt LASSO vs LASSO
In the paper Square Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming they talk about Sqrt-LASSO which is simply just trying to minimize $\|Ax-b\|_2 + \lambda\|x\|_1$ rather than ...
- 63