Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [linear-regression]

For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.

0
votes
0answers
10 views

Instrumental variable exogeneity

Suppose $\ Y = a + bX + U, cov(X,U) = 0$ Define $\ Z = mX$ where m is a non-zero constant Can Z be used as an instrumental variable? - as Z = mX, the Z is relevant but is Z uncorrelated with U?
0
votes
2answers
20 views

Including the (0,0) point in regression

I have run a simple linear regression in Rstudio with two variables and got the following relation: y = 30000+1.95x Which is reasonably fair. My only concern is that, practically the (0,0) point ...
0
votes
1answer
20 views

Logistic Regression Explanation

I have two questions regarding logistic regression. 1) I understand that the results of a logistic regression model yield a table stating coefficients together with a p-statistic for each variable . ...
0
votes
2answers
19 views

Linear Regression Assumption

the following website states that linear regression assumes a linear relationship between dependent and independent variables: "First, linear regression needs the relationship between the independent ...
0
votes
0answers
10 views

Calculating the Standard Error of OLS estimates

So, my understanding of standard error is the following: Suppose I have a population of heights of individuals. From this population I draw samples of size n = ...
0
votes
1answer
28 views

Linear Regression Assumption: Normality of residual vs normality of variables

I have read in many places, including stack exchange, that in order to carry linear regression analysis the residuals have to be normal. This is required because most of the statistical results, ...
0
votes
0answers
13 views

Operating with penalized regression models

Assume we are working with a penalized linnear regression model. We have the following optimization problem: \begin{equation} \min_{\beta}\left\{\left\lVert y-X\beta\right\rVert_2^2+\lambda\sum_{j=1}^...
0
votes
1answer
20 views

Proof of Batch Gradient Descent's cost function gradient vector

In the book Hands-On Machine Learning with Scikit-Learn & TensorFlow, the author only showed the formula for the Batch Gradient Descent method, such as: $ \dfrac{\partial}{\partial \theta_{j}} ...
1
vote
1answer
41 views

Partial derivative of MSE cost function in Linear Regression?

I'm confused by multiple representations of the partial derivatives of Linear Regression cost function. This is the MSE cost function of Linear Regression. Here $h_\theta(x) = \theta_0+\theta_1x$ . \...
1
vote
0answers
19 views

What is saturated model used in glm in R?

So, I've read on stack exchange (glm summary explained) that residual deviance computed in the glm output is just the likelihood ratio chi-square stat comparing the saturated model to the reduced ...
1
vote
0answers
16 views

How to find the correspondence between ridge coefficients to modified ridge?

The ridge coefficients minimize a penalised residual sum of squares: $\hat{w}_{ridge}= \arg\min(\sum_{i=1}^{N}(y_{i}-w_{0}-\sum_{j=1}^{d}x_{ij}w_{j})^{2}+\lambda\sum_{j=1}^{d}w_{j}^{2}),\,\lambda\...
0
votes
1answer
12 views

Effect on $R^2$ squared of an additional regressor

I'm currently dealing with the simple linear regression model and the book I'm studying with says that, any time you add a regressor to your model, even if irrelevant, the coefficient of determination ...
0
votes
0answers
14 views

Multiple Regression Assumption

If I am running a multiple linear regression model with six independent variables against dependent variable, do the assumptions of multiple regression need to be satisfied? or does this only applies ...
0
votes
2answers
30 views

Formulating the polynomial regression

I'm trying to formulate a regression problem such that $y=ax^{b}$. Previously, I formulated the $y=ax+b$ like $y=Ac+e$ where $c= \begin{bmatrix} a\\ b \end{bmatrix}$ and $A= \begin{bmatrix} x_1 &...
0
votes
0answers
25 views

3d linear regression line equations

I have a bunch of points in 3D space (around to a survey line). Since only half of the point coordinates were measured, I want to compute a linear regression, from which I extrapolate the other points ...
1
vote
0answers
16 views

Regressions with two independent variables where one is part of another.

I am doing simple regression analysis where I use one-way fixed effect model to estimate the effects of two variables on dependent variable. The question I am asking is how to interpret these two ...
0
votes
0answers
22 views

Fitting a Straight Line to 3D Points, and then finding the nearest point to those lines.

I've been stuck on this problem for over a week now. Essentially, I have a group of 4 tracks of points from a bunch of detectors in 3 dimensions. I need to map these points to straight lines and then ...
1
vote
0answers
15 views

How to calculate $\text{cov}(\hat{Y}_{ij}, \hat{Y}_{kj})$ if $Y_{ij} = \mu + a_i + b_j + e_{ij}$?

Let's assume we have the model following Two-Factor model without replications : $$Y_{ij} = \mu + a_i + b_j + e_{ij}, \; i=1,\dots,p \; \text{and} \; j=1,\dots, q $$ I am interested in calculating ...
2
votes
0answers
17 views

Covariance of errors $\text{cov}(\hat{e_{ij}},\hat{e_{i\ell}})$ in Two-Way Anova model

Exercise : Consider the Two-Way Anova model $Y_{ij} = \mu + a_i + b_j + e_{ij}$ with $i = 1, \dots, p$ and $j=1,\dots,q$. Show that : $$\text{cov}(\hat{e_{ij}},\hat{e_{i\ell}}) = -\sigma^2\left(\...
1
vote
1answer
41 views

fitting triple exponential term function to data

The function, I am trying to fit to data is: $$y(x) = −(A+B)e^{−x/a_1} + 𝐴e^{−x/a_2} + Be^{−x/a_3}$$ this function is a little bit different to Is it possible to find initial parameters when fitting ...
0
votes
1answer
17 views

question about matrix equation for coefficients in linear regression

There is a matrix equation for solving a linear regression, $\vec{y}=X\vec{\beta}$ where $X$ is the matrix of features, $\vec{\beta}=[\beta_1,...,\beta_n]$ are the coefficients for each feature, and $\...
0
votes
1answer
14 views

Interpreting linear regression coefficients for a covariate that's correlated with other covariates

The interpretation of linear regression coefficients that I learned is that the coefficient is the change in outcome associated with a unit change in that covariate, assuming all other covariates stay ...
0
votes
1answer
23 views

Proving least squares is a minima

Minimizing $ f(\beta) = (Y-X\beta)'(Y-X\beta) $ we take the derivative set it to zero and get $ \beta = (X'X)^{-1}X'Y $ now everywhere I've read they say this is the minima because $ (X'X) $ is ...
0
votes
0answers
22 views

Multiple Linear Regression: what to do with $\hat{\beta}_i$

I am working on a problem where this is the given data. I ran an MLR and it gave me the following result on R. ...
0
votes
0answers
11 views

Standard error of $\hat{\beta_1}$ in regression analysis

In the earlier chapters of my notes, the formula for $\hat{\beta_1}$ in simple linear regression was given as $$\frac{\hat{\sigma}}{\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2}}$$. However, in some later ...
0
votes
1answer
11 views

Ridge Regression Coefficient Estimate is linear

I was able to derive the formula for the Ridge Regression Coefficient Estimate $\hat{\boldsymbol{\beta}}^{ridge}$. However, I am not 100% sure what it means in terms of showing that the Ridge ...
1
vote
1answer
41 views

Stupid question but unsure on Regression.

When I want to do a regression between daily change of currency prices and a daily stock index changes which one shall I use as the dependent and independent variable. Thanks for any help.
1
vote
1answer
22 views

Regression model + expected value, variance and autocorrelation of the error term

Consider this regression model $$Y_t=X_t\beta+\epsilon_t, ~~~~~~~~~~\epsilon_t \sim WN(0, \sigma^2_{\epsilon})$$ with 3 different specifications of the error term: $\epsilon_t=\alpha_1\epsilon_{t-1}+...
1
vote
1answer
35 views

Linear regression - find $b$, $s^2$, $R^2$

Suppose you want to fit the model $Y=\alpha+\beta X+\epsilon$ but you don't have the full data set $\left[\begin{matrix}y&X\\\end{matrix}\right]=C$. Instead you only have: $$C'C=\begin{bmatrix} ...
0
votes
0answers
7 views

Why is scatter plot for $Y$ against $\hat{Y}$ unnecessary for SLR?

I am still new to this, so I apologize if my question is conceptually wrong. What I understand is that we sometimes need to use the scatter plot to find out if there is a relationship, besides the ...
0
votes
0answers
26 views

Expectation of random variable with omitted variable bias

I am given: $y_i = \beta_0 + \beta_1range_i + \beta_2dist_i + \epsilon_i$ $E[\epsilon_i | range_i, dist_i]=0$ Then in the case where $dist_i$ is omitted: $y_i = \beta_0 + \beta_1range_i + \eta_i$ ...
1
vote
3answers
42 views

Linear Regression and Ridge Regression

In Linear regression, we have $\textbf{$\theta$} = \textbf{$\left (X^TX\right )^{-1} X^T y$}$. In Ridge regression, we have $\textbf{$\theta$} = \textbf{$\left ( \lambda I+ X^TX\right )^{-1} X^T y$}$...
0
votes
1answer
17 views

Feature matrix as the Kronecker product of two feature matrices. How to build an alternative?

I have two feature matrices $\textbf{X}$ and $\textbf{Y}$ which I encoded through one-hot encoding the rows of two feature matrices $\textbf{X'}$ and $\textbf{Y'}$. Thus, they are sparse with a few $1$...
0
votes
1answer
16 views

Algebraic transformation — where is my mistake?

I tried to find the estimators of $\hat{\beta_1}$ and $\hat{\beta_0}$ via the least-squares method algebraically. Somehow I seem to have messed up. Can you tell me where? My Calculations.
1
vote
1answer
48 views

How to derive the variance of the mean of predictions from a linear regression model?

The context is linear regression analysis for estimating a sample mean. Assume the usual multivariate linear model: $$Y = X\beta + \epsilon$$ with $X$ a $n \times p$ covariate matrix with intercept, $...
0
votes
0answers
38 views

Textbook to learn one-way and two-way ANOVA

I'm looking for a good (detailed, mathematical) textbook to learn one-way and two-way ANOVA. I'm new to statistics and I was learning linear regression from D. C. Montgomery, E. A. Peck, G. G. Vining,...
0
votes
0answers
18 views

Relationship between different types of correlation coefficients

Let, $r_{1(2.34...p)}$ = Correlation between $x_1$ and $x_{2.34...p}$. The latter being the residuals after regressing $x_2$ on $x_3 , x_4 ....x_p$. $r_{1.234..p}$ = Multiple correlation coefficient ...
0
votes
1answer
16 views

Calculating $\hat{\beta_1}$ given $\overline{X}$, $\overline{Y}$, their standard deviations and the correlation coefficient.

In a linear regression problem I'm asked to find $\hat{\beta_1}$ given only the following information: $\overline{X} = 163.5$ $\overline{Y} = 874.1$ $\sigma_X = 16.2$ $\sigma_Y = 54.2$ $r = -0.774$ ...
1
vote
0answers
17 views

Converting back to unnormalized domain after normalizing

I'm trying to perform a linear regression on a dataset however, without normalizing the data I couldn't achieve any results therefore I was forced to normalize it. Now my graph looks fine for the ...
0
votes
0answers
14 views

compare Bayesian linear regression vs standard linear regression

1st question, I recently learnt bayesian linear regression, but I'm confused that in what situation we should use bayesian linear regression, and when to use standard linear regression? What is the ...
0
votes
1answer
52 views

Help in understanding Matrix Differentiation laws used in Stochastic Gradient Descent

I come from a programming background. I am familiar with scalar calculus but not so much with vector/matrix calculus. I am trying to understand stochastic gradient descent for multiple linear ...
0
votes
0answers
11 views

how write residual by adding outliers?

I'm reading a paper about outliers and its detection. there they suppose a initial group of observatios and next they add a group of high leverage identical outliers. where to the initial observations ...
1
vote
1answer
35 views

how to bound the hat matrix?

I'm reading a paper about linear regression and in some point they define: $$ w_{ij}=\frac{h^{2}_{ji}}{ph_{ii}(1-h_{jj})^{2}}$$, where $h_{ij}$ are the elements of the hat matrix. The problem is ...
0
votes
1answer
39 views

Multivariate Linear regression with fewer trained parameters

I have a multivariate linear regression problem to solve for identifying a dynamic greybox model. Normally I would formulate it in this form: Given $n$ number of observations, $m$ response variables,...
2
votes
1answer
62 views

Isotonic regression with a linear constraint

I'm trying to find a direct approach to solving (for some fixed vector $y$): $$ \begin{aligned} \min & \; \|x - y \|^2 \\ \mbox{s.t. } & \alpha^\top x \leq 0 \\ & x_i \...
0
votes
1answer
24 views

Calculating error on slope of graph

I'm trying to find the rate of change and the error on that rate based on 7 measurements points and the assumption that the trend is linear. My calculations are below: $$ \begin{array}{|c|c|c|c|} \...
0
votes
0answers
9 views

How find the Cook's distance expected value for a large n

I'm trying to prove that for a large n, the Cook expected value is approximated by: $$E[D_{i}]\approx \frac{h_{ii}}{p(1-h_{ii})}$$. Do you know where can i read the prove of that?. i tried to do it ...
0
votes
0answers
12 views

How do I derive a Least Square Estimator of a linear combination of two variables?

I am working on a problem where I have the following model: lm(Y ~ x1 x2) If I have the output of this general model in R, is it possible to derive the LSE of: <...
0
votes
1answer
31 views

line nearest to a set of points

In stack overflow there are at least two questions asking how to find the straight line in a 2D space nearest to a given set of points $\vec{a}_i$ for $i = 1...N$: question 1 question 2. In both ...
0
votes
1answer
42 views

what is the -(2/n) in the gradient descent

I am trying to explore how gradient descent works. I found a Formula $$\begin{aligned} \frac { \partial } { \partial \mathrm { m } } & = \frac { 2 } { N } \sum _ { i = 1 } ^ { N } - x _ { i }...