Questions tagged [linear-regression]

For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Express the regularized weight in ridge regression in terms of the linear regression solution .

We would like to minimize the quantity $E_{in}(\vec{w})=\frac{1}{N}\sum_{i=1}^N(\vec{w}^{T}\vec{x_n}-y_n)^2$ under the constraint $\vec{w}^T\Gamma^T\Gamma\vec{w}\leq C$ where $\Gamma$ is a matrix, $C$ ...
lcthaha's user avatar
1 vote
0 answers
55 views

Is the expectation of the error of any projection of $\mathbb{E}[Y\mid X]$ onto subspace zero?

If we consider the following linear predictor of $Y$ based on $X$: $$ Y_{\mathbf{b}}=\boldsymbol{\Sigma}_{Y, \mathbf{X}} \boldsymbol{\Sigma}_{\mathbf{X}}^{-1}\left(\mathbf{X}-\boldsymbol{\mu}_{\mathbf{...
maskeran's user avatar
  • 521
0 votes
0 answers
10 views

Finding parameters for function approximation

I am working on some project in Matlab, where I defined some function $R(x, h, H, L, N)$ and I want to find such $h, H, L, N$ such that $R(x, h, H, L, N)$ is approximated by some sine wave in other ...
Uroš Kosmač's user avatar
0 votes
0 answers
10 views

closed form expression for training error in ridge regression

I'm reading the paper a random matrix approach to neural network but i'm stucked at page 5. They start from $\Sigma \in \mathbb{R}^{n\times T}$ where $T$ is the number of data points and $n$ is the ...
fabianod's user avatar
  • 155
0 votes
1 answer
161 views

Can a (bounded) linear least-squares problem include a scale factor in its solution?

I have a system of equations. $$ \small \begin{aligned} (1 - p_0)u_0 + (q_0 - 1)v_0 + u_1 - v_1 = r_0 - c t_0 \\\\ (1 - p_1)u_1 + (q_1 - 1)v_1 + u_0 - v_0 = r_1 - c t_1 \\\\ (1 - p_2)u_2 + (q_2 - 1)...
Isco's user avatar
  • 93
0 votes
0 answers
17 views

Derivation of normal equation for linear regression parameters

I'm going through a derivation of the normal equation for the parameter vector $\beta$ of the linear regression model. Given a model $y = X\beta + \epsilon$, where $y$ is the vector of dependent ...
Riccardo Iorio's user avatar
0 votes
0 answers
42 views

Is conditional expectation of the error of best linear predictor given $X$ is $0$ (Is it true that $y = a^*+b^*x + \eta$, where $E[\eta|x]=0$)?

For simplicity, assume we are working with simple regression where the predictor $x\in\mathbb{R}$. First write $y=E[y \mid x]+u$, where the variance of $u$ is a constant, and $E[u|x]=0$. I understand $...
maskeran's user avatar
  • 521
0 votes
0 answers
13 views

Results of linear regression with just statistically significant features vs. whole dataset

If I have a dataset, in this case the diabetes toy dataset, and am running a linear regression model, could someone explain what I should expect in terms of performance if I were to conduct the ...
InvestingScientist's user avatar
0 votes
0 answers
21 views

PDF Being Invariant Under Orthogonal Transformations

Regarding linear regression, in the (informal) solution manual of T. Hastie's "The Elements of Statistical Learning" book, there is a paragraph (p. 16 in the manual) that goes: "... ...
ConventionalProgrammer's user avatar
0 votes
1 answer
39 views

Population OLS coefficient in simple regression?

The population OLS coefficient for some $X_i \in \mathbb{R}^d, Y \in \mathbb{R}$ for the model $Y = \beta’X + e$ is defined as $$ \beta =\mathbb{E}[X_iX_i']^{-1}\mathbb{E}[X_iY_i] $$ and if $X$ is a ...
tmako's user avatar
  • 196
2 votes
2 answers
185 views

Is there a rigorous probability theoretic formulation of linear regression

Let $(\Omega,\Sigma,P)$ be some probability space and let $Y: \Omega \to \mathbb{R} $ be a random variable. Let $x, \beta \in \mathbb{R}^n$. In a linear model we assume something like this: $E(Y|x)=\...
Bipolar Minds's user avatar
0 votes
0 answers
17 views

Creating a regression model of scaling data from five points for ATAR

I am looking to calculate the scaled results for the ATAR (Australian Tertiary Admission Rank) subjects that someone inputs from the recently released 2023 data. For example if someone got a 59.00 in ...
Wosca's user avatar
  • 3
0 votes
0 answers
35 views

Simpson's Paradox for $R^2$

I've been reading about Simpson's Paradox and I understand how if you fit a regression on a per-group basis versus on the entire set, the coefficients in each group can be drastically different from ...
jeffj1355's user avatar
0 votes
0 answers
18 views

Convergence of coefficients in multivariate regression

In this thread, the convergence of coefficient for univariate dependent variable is proven. I wonder, assuming the same setup, how can the convergence be extended to multivariate as: $$Y=XW+\epsilon$$ ...
statwoman's user avatar
  • 253
0 votes
0 answers
48 views

Proving product of two projection matrices is commutative

I’ve been trying to understand the details of orthogonal projections within the context of linear regression models, and I’ve come across the following scenario: Given a linear regression model $Y=\...
Noy's user avatar
  • 777
3 votes
0 answers
49 views

Sqrt LASSO vs LASSO

In this paper they talk about Sqrt-LASSO which is simply just trying to minimize $\|Ax-b\|_2 + \lambda\|x\|_1$ rather than the regular LASSO $\|Ax-b\|_2^2 + \lambda\|x\|_1$. Can anyone point out the ...
jeffj1355's user avatar
-1 votes
1 answer
31 views

Lognormal linear regression

I would like to fit a linear regression model with a lognormal distribution that has a linear expectation in the $X$, $Y$ space. So, I need a model with the following properties: $$ Y|X \sim Lognormal ...
Amav's user avatar
  • 3
0 votes
1 answer
35 views

Lagrange Interpolation as generalized polynom

I need to write an algorithm that constructs a function of the form $f(x) = \sum_i^n q_i x^i$ that exactly goes through the points $p_i = (a_i, b_i)$. In general building such a function is not the ...
Sprinklerkopf's user avatar
0 votes
0 answers
19 views

Property of projection matrix $P = X(X'X)^{-1}X'$ under random design context

In OLS with fixed design, we project $Y$ onto the column space of design matrix $X$, which is $\mathcal{C}(X)$. The residual $Y-PY = (I-P)Y$ is orthogonal $\mathcal{C}(X)$. Further more, $(I-P)$ is a ...
maskeran's user avatar
  • 521
0 votes
1 answer
33 views

Can a single dummy variable be made to meet multiple criteria?

Let say I have criteria $1, 2, 3,$ and $4$. I would like the Dummy variable to be 1 only if a certain minimum amount of criteria are met. For example, if $3$ of the $4$ are true, then Dummy$ =1$. If $...
MLux's user avatar
  • 1
1 vote
1 answer
60 views

P-value change for dropping orthogonal explanatory variable

Consider a linear model $X\beta + \epsilon$, where $E(\epsilon) = 0$ and a fixed deterministic $n\times p$ design matrix $X$. $\beta = (\beta_1, ..., \beta_p)^T$, $rank(X) = p$ Explain wheter the ...
y4nik's user avatar
  • 193
3 votes
2 answers
195 views

How to fit and ODE to data?

Consider the following ODE $$ y'(t)=\alpha x(t)-\beta y(t) $$ and the following datasets $$ X=\{(t_0,x_0),...,(t_n,x_n)\}\\ Y=\{(t_0,y_0),...,(t_n,y_n)\} $$ How can I find $\alpha$ and $\beta$ that ...
sam wolfe's user avatar
  • 3,335
1 vote
2 answers
36 views

Linear regression prediction equal to data when using normal equation?

So I am currently re-studying linear regression and wanted to explore the topic more thoroughly than before. The prediction formula for linear regression is: $$\hat Y = X \hat \theta $$ Now, the ...
Karol Szustakowski's user avatar
0 votes
0 answers
27 views

Estimation of the variance of ML estimator (linear regression)

Given the following likelihood: $$\prod_{i=1}^{N} \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left\{-\frac{(y_i - x_i' \beta)^2}{2\sigma^2}\right\}$$ Thanks to the information matrix equality we have two ...
Marlon Brando's user avatar
-1 votes
1 answer
68 views

Matrix product in linear regression [closed]

I do not understand why $(X^TX)^{-1}X^T * (X^TX)^{-1}X^T = (X^TX)^{=1}*I$ I keep trying to work it out and I just don't see it. This equality shows up when deriving the variance formula for ...
AdmiralMunson's user avatar
0 votes
1 answer
32 views

Question about finding slope from my textbook, walking through linear regression through the least squares method (pre-calculus)

I have question about a specific step describing how to do linear regression with the least squares method. In the textbook, it says that you need to find $\triangle x$ and $\triangle y$ (which are $...
B Karr's user avatar
  • 1
1 vote
0 answers
36 views

Deriving formula for sum of squares regression

I measured some 2D data, which when graphed follows a linear function. I want to fit this function and find the parameters $a$ and $b$ that best match the data. I am now trying to derive the formula ...
Adam Labuš's user avatar
0 votes
0 answers
37 views

Application of the Generalized Method of Moments

The professor gave us an exercise but for me it's not totally clear. I have a distribution function which depend on a parameter $x$ that I need to estimate. The function describes the distribution of ...
Arianna's user avatar
0 votes
1 answer
77 views

Derivation of $SSE =S_{yy} - S_{xy}^2/S_{xx}$

I have seen this result and I am trying to figure out how to derive it from $SSE = \sum(Y_i - \hat{Y})^2$. I know that $r = \frac{s_{xy}}{\sqrt{s_{xx}s_{yy}}} $ and I have seen online to use and ...
Jackanap3s's user avatar
0 votes
0 answers
38 views

Same result in every iterations from subgradient and proximal gradient method.

I'm trying to implement the subgradient method and proximal gradient method with constant stepsize for the lasso problem but the result for the subgradient method and proximal gradient is almost ...
Help me pls's user avatar
0 votes
0 answers
22 views

Obtaining multivariate linear regression coefficients from residual values of simple linear regressions

We are interested in a multivariate linear regression of $Y$ against $X_1$ and $X_2$. More specifically we want to know the regression coefficients of $X_1$. There is a sample drawn of size $n$, but ...
Uberfatty's user avatar
  • 141
0 votes
0 answers
33 views

Distinguishing various types of regression

I want to understand the pairwise relationship between four types of regression: Bayesian Linear Regression, Gaussian Process Regression, Kernel Regression (Nadaraya-Watson), and Kernel Ridge ...
Tanishq Kumar's user avatar
0 votes
1 answer
26 views

Linear independence as a way to gauge predictor usefulness.

Background. The multiple linear regression model is of the form $$ Y = \beta_0 + \beta_1X_1 + \cdots + \beta_nX_n + \epsilon $$ where we assume $\epsilon$ is normally distributed with constant ...
Paul Ash's user avatar
  • 1,131
1 vote
0 answers
45 views

$l_{\infty}$ convergence of OLS prediction error

I have found many resources talking about the $l_2$ convergence of OLS estimation error and prediction error. I have also read about the analysis of the $l_2$ convergence of lasso estimation error and ...
maskeran's user avatar
  • 521
2 votes
2 answers
97 views

Least relative error linear regression solution

Consider the optimization problem of minimizing the function $f(a, b)$ defined as: $$f(a, b) = \sum_{i=1}^n \left(\frac{x_i - \frac{y_i - b}{a}}{x_i}\right)^2,$$ where $\{(x_1, y_1), (x_2, y_2), \...
Ray Bern's user avatar
  • 627
0 votes
1 answer
32 views

Minimizing error with several linear regressions and choosing the best one for each point

Given $n$ points $(x_1, y_1), (x_2, y_2), \ldots, (x_n ,y_n)$ and $m \in \mathbb N$. We aim to find $m$ linear models $f_j : y = k_j x + b_j, j \in [m]$ to minimize $\sum_{i \in [n]} \min_{j \in [m]} ...
Vezen BU's user avatar
  • 1,961
0 votes
1 answer
20 views

Residual Normality Test

I did linear regression for 3 variables from 60 data. So, my equation is: $$Y = \alpha_0 + \alpha_1X_1 + \alpha_2X_2 + \alpha_3X_3.$$ One of the test is residual-normality test. Can I just use Central-...
Niccolo's user avatar
  • 694
0 votes
0 answers
16 views

Is OLS estimator sub-Gaussian?

Suppose we have $n$ observations $(x_i,y_i)\in\mathbb{R}^{p+1}$. Suppose the underlying model is $y = \eta^{*T} x + \varepsilon$, and $x, \varepsilon$ are sub-Gaussian with parameter $\sigma_x, \...
maskeran's user avatar
  • 521
0 votes
0 answers
84 views

The canonical link function of a corresponding generalized linear model.

Consider a discrete probability distribution with the following probability mass function $f(y;\lambda ,\nu) =\frac{\lambda^y}{A(\lambda,\nu)(y!)^\nu}$, $y= 0,1,2,...,$ with parameters $\lambda \gt ...
User1's user avatar
  • 31
2 votes
0 answers
51 views

I'm a bit confused on how to linearize an Exponential Graph

I've seen other people answer questions on how to do this, I tried it and it didn't work. As of right now, I'm using the following desmos graphs. Desmos, Exponential Graph Desmos, Linearized ...
Yashwak's user avatar
  • 15
0 votes
1 answer
115 views

weighted least squares and variance stabilizing transformation

Consider a simple linear regression model with response $y$, an intercept and an explanatory variable $x$,i.e. $$y_i=\beta_0+\beta_1x_i+\epsilon_i, \;\;\;\;\;\;\;\; i=1,...,n.$$ Assume further that ...
User1's user avatar
  • 31
0 votes
0 answers
16 views

Orthogonal Projection - Uncorrelatedness & Orthogonality

Can we assume that the correlation of the residuals $û$ from an OLS regression with the explanatory variables is always equal to zero? The orthogonal projection matrix is given as $P = X (X'X)^{-1} X'$...
Marlon Brando's user avatar
1 vote
1 answer
53 views

How to solve a linear ridge regression.

The below is some part of this thesis. https://kaiminghe.github.io/publications/pami12guidedfilter.pdf. I could not understand how equation (4) is calculated. I know that I need to do “partial ...
CamFly's user avatar
  • 31
0 votes
1 answer
32 views

Linear $\log$ models change in betas

I'm trying to solve an exercise but I find it difficult to interpret. I have a linear-$\log$ model like this: $y = 1 + 0.55\ln(x) + 3z - 2.2w + \text{error term}$. I wonder what happens to $\beta_1$ ...
Alessandro Tursilli's user avatar
2 votes
1 answer
38 views

Deriving OLS Estimation with Measurement Error

I am trying to assess the bias of a regression model such that the true model, $y=X\beta+u$ is amended such that the true $X$ is replaced with $\widetilde{X} = X + \epsilon$, where $\epsilon$ is a ...
Kate's user avatar
  • 21
2 votes
0 answers
61 views

GLMs for exponential families - why is the canonical link invertible?

I am studying GLMs for exponential families and wondering how the density of exponential families gives rise to a canonical link function. Following the notation from Fahrmeier (Regression), the ...
AdaLovelace's user avatar
0 votes
1 answer
104 views

variance-stabilizing transformation on a simple linear regression model

Consider a simple linear regression model with response $y$, an intercept and an explanatory variable $x$,i.e. $$y_i=\beta_0+\beta_1x_i+\epsilon_i, \;\;\;\;\;\;\;\; i=1,...,n.$$ Assume further that ...
User1's user avatar
  • 31
1 vote
1 answer
29 views

Linear regression, what happens to new betas when they're multiplied by a scalar

Suppose that: Y = 1 + 5X + ϵ We multiply Y by 3 and X by 1/4 and compute a new regression model: find the values of β0, and β1. Hi guys, I'm trying to solve this question my thinking leads me to solve ...
Alessandro Tursilli's user avatar
0 votes
0 answers
22 views

Link functions for Gaussian process priors

I am working through the following paper and am wondering about the link function considered there. I am referring to Definition 7 and Example 8 in https://arxiv.org/pdf/1809.08818.pdf. Why is it ...
melmo99's user avatar
  • 107
0 votes
1 answer
56 views

Independence of pure error and lack-of-fit error in simple linear regression with repeated observations

Let $x_1,\ldots,x_n$ be distinct regressor variables. For each $x_i$, there are $n_i$ observations $Y_{i1},\ldots,Y_{in_i}$ such that $$Y_{ij}=\alpha x_i+\beta+\epsilon_{ij},$$ where $\epsilon_{ij}\...
ashpool's user avatar
  • 6,946

1
2 3 4 5
27