Questions tagged [linear-regression]
For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.
0
votes
0answers
10 views
Instrumental variable exogeneity
Suppose $\ Y = a + bX + U, cov(X,U) = 0$
Define $\ Z = mX$ where m is a non-zero constant
Can Z be used as an instrumental variable?
- as Z = mX, the Z is relevant but is Z uncorrelated with U?
0
votes
2answers
20 views
Including the (0,0) point in regression
I have run a simple linear regression in Rstudio with two variables and got the following relation:
y = 30000+1.95x
Which is reasonably fair. My only concern is that, practically the (0,0) point ...
0
votes
1answer
20 views
Logistic Regression Explanation
I have two questions regarding logistic regression.
1) I understand that the results of a logistic regression model yield a table stating coefficients together with a p-statistic for each variable . ...
0
votes
2answers
19 views
Linear Regression Assumption
the following website states that linear regression assumes a linear relationship between dependent and independent variables:
"First, linear regression needs the relationship between the independent ...
0
votes
0answers
10 views
Calculating the Standard Error of OLS estimates
So, my understanding of standard error is the following:
Suppose I have a population of heights of individuals. From this population I draw samples of size n = ...
0
votes
1answer
28 views
Linear Regression Assumption: Normality of residual vs normality of variables
I have read in many places, including stack exchange, that in order to carry linear regression analysis the residuals have to be normal. This is required because most of the statistical results, ...
0
votes
0answers
13 views
Operating with penalized regression models
Assume we are working with a penalized linnear regression model. We have the following optimization problem:
\begin{equation}
\min_{\beta}\left\{\left\lVert y-X\beta\right\rVert_2^2+\lambda\sum_{j=1}^...
0
votes
1answer
20 views
Proof of Batch Gradient Descent's cost function gradient vector
In the book Hands-On Machine Learning with Scikit-Learn & TensorFlow, the author only showed the formula for the Batch Gradient Descent method, such as:
$ \dfrac{\partial}{\partial \theta_{j}} ...
1
vote
1answer
41 views
Partial derivative of MSE cost function in Linear Regression?
I'm confused by multiple representations of the partial derivatives of Linear Regression cost function.
This is the MSE cost function of Linear Regression. Here $h_\theta(x) = \theta_0+\theta_1x$ .
\...
1
vote
0answers
19 views
What is saturated model used in glm in R?
So, I've read on stack exchange (glm summary explained) that residual deviance computed in the glm output is just the likelihood ratio chi-square stat comparing the saturated model to the reduced ...
1
vote
0answers
16 views
How to find the correspondence between ridge coefficients to modified ridge?
The ridge coefficients minimize a penalised residual sum of squares:
$\hat{w}_{ridge}= \arg\min(\sum_{i=1}^{N}(y_{i}-w_{0}-\sum_{j=1}^{d}x_{ij}w_{j})^{2}+\lambda\sum_{j=1}^{d}w_{j}^{2}),\,\lambda\...
0
votes
1answer
12 views
Effect on $R^2$ squared of an additional regressor
I'm currently dealing with the simple linear regression model and the book I'm studying with says that, any time you add a regressor to your model, even if irrelevant, the coefficient of determination
...
0
votes
0answers
14 views
Multiple Regression Assumption
If I am running a multiple linear regression model with six independent variables against dependent variable, do the assumptions of multiple regression need to be satisfied?
or does this only applies ...
0
votes
2answers
30 views
Formulating the polynomial regression
I'm trying to formulate a regression problem such that $y=ax^{b}$. Previously, I formulated the $y=ax+b$ like $y=Ac+e$ where $c=
\begin{bmatrix}
a\\
b
\end{bmatrix}$ and $A=
\begin{bmatrix}
x_1 &...
0
votes
0answers
25 views
3d linear regression line equations
I have a bunch of points in 3D space (around to a survey line). Since only half of the point coordinates were measured, I want to compute a linear regression, from which I extrapolate the other points ...
1
vote
0answers
16 views
Regressions with two independent variables where one is part of another.
I am doing simple regression analysis where I use one-way fixed effect model to estimate the effects of two variables on dependent variable. The question I am asking is how to interpret these two ...
0
votes
0answers
22 views
Fitting a Straight Line to 3D Points, and then finding the nearest point to those lines.
I've been stuck on this problem for over a week now.
Essentially, I have a group of 4 tracks of points from a bunch of detectors in 3 dimensions. I need to map these points to straight lines and then ...
1
vote
0answers
15 views
How to calculate $\text{cov}(\hat{Y}_{ij}, \hat{Y}_{kj})$ if $Y_{ij} = \mu + a_i + b_j + e_{ij}$?
Let's assume we have the model following Two-Factor model without replications :
$$Y_{ij} = \mu + a_i + b_j + e_{ij}, \; i=1,\dots,p \; \text{and} \; j=1,\dots, q $$
I am interested in calculating ...
2
votes
0answers
17 views
Covariance of errors $\text{cov}(\hat{e_{ij}},\hat{e_{i\ell}})$ in Two-Way Anova model
Exercise :
Consider the Two-Way Anova model $Y_{ij} = \mu + a_i + b_j + e_{ij}$ with $i = 1, \dots, p$ and $j=1,\dots,q$. Show that :
$$\text{cov}(\hat{e_{ij}},\hat{e_{i\ell}}) = -\sigma^2\left(\...
1
vote
1answer
41 views
fitting triple exponential term function to data
The function, I am trying to fit to data is:
$$y(x) = −(A+B)e^{−x/a_1} + 𝐴e^{−x/a_2} + Be^{−x/a_3}$$
this function is a little bit different to Is it possible to find initial parameters when fitting ...
0
votes
1answer
17 views
question about matrix equation for coefficients in linear regression
There is a matrix equation for solving a linear regression, $\vec{y}=X\vec{\beta}$ where $X$ is the matrix of features, $\vec{\beta}=[\beta_1,...,\beta_n]$ are the coefficients for each feature, and $\...
0
votes
1answer
14 views
Interpreting linear regression coefficients for a covariate that's correlated with other covariates
The interpretation of linear regression coefficients that I learned is that the coefficient is the change in outcome associated with a unit change in that covariate, assuming all other covariates stay ...
0
votes
1answer
23 views
Proving least squares is a minima
Minimizing $ f(\beta) = (Y-X\beta)'(Y-X\beta) $ we take the derivative set it to zero and get $ \beta = (X'X)^{-1}X'Y $ now everywhere I've read they say this is the minima because $ (X'X) $ is ...
0
votes
0answers
22 views
Multiple Linear Regression: what to do with $\hat{\beta}_i$
I am working on a problem where this is the given data.
I ran an MLR and it gave me the following result on R.
...
0
votes
0answers
11 views
Standard error of $\hat{\beta_1}$ in regression analysis
In the earlier chapters of my notes, the formula for $\hat{\beta_1}$ in simple linear regression was given as $$\frac{\hat{\sigma}}{\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2}}$$.
However, in some later ...
0
votes
1answer
11 views
Ridge Regression Coefficient Estimate is linear
I was able to derive the formula for the Ridge Regression Coefficient Estimate $\hat{\boldsymbol{\beta}}^{ridge}$. However, I am not 100% sure what it means in terms of showing that the Ridge ...
1
vote
1answer
41 views
Stupid question but unsure on Regression.
When I want to do a regression between daily change of currency prices and a daily stock index changes which one shall I use as the dependent and independent variable. Thanks for any help.
1
vote
1answer
22 views
Regression model + expected value, variance and autocorrelation of the error term
Consider this regression model
$$Y_t=X_t\beta+\epsilon_t, ~~~~~~~~~~\epsilon_t \sim WN(0, \sigma^2_{\epsilon})$$
with 3 different specifications of the error term:
$\epsilon_t=\alpha_1\epsilon_{t-1}+...
1
vote
1answer
35 views
Linear regression - find $b$, $s^2$, $R^2$
Suppose you want to fit the model $Y=\alpha+\beta X+\epsilon$ but you don't have the full data set $\left[\begin{matrix}y&X\\\end{matrix}\right]=C$. Instead you only have:
$$C'C=\begin{bmatrix}
...
0
votes
0answers
7 views
Why is scatter plot for $Y$ against $\hat{Y}$ unnecessary for SLR?
I am still new to this, so I apologize if my question is conceptually wrong. What I understand is that we sometimes need to use the scatter plot to find out if there is a relationship, besides the ...
0
votes
0answers
26 views
Expectation of random variable with omitted variable bias
I am given:
$y_i = \beta_0 + \beta_1range_i + \beta_2dist_i + \epsilon_i$
$E[\epsilon_i | range_i, dist_i]=0$
Then in the case where $dist_i$ is omitted:
$y_i = \beta_0 + \beta_1range_i + \eta_i$
...
1
vote
3answers
42 views
Linear Regression and Ridge Regression
In Linear regression, we have $\textbf{$\theta$} = \textbf{$\left (X^TX\right )^{-1} X^T y$}$.
In Ridge regression, we have $\textbf{$\theta$} = \textbf{$\left ( \lambda I+ X^TX\right )^{-1} X^T y$}$...
0
votes
1answer
17 views
Feature matrix as the Kronecker product of two feature matrices. How to build an alternative?
I have two feature matrices $\textbf{X}$ and $\textbf{Y}$ which I encoded through one-hot encoding the rows of two feature matrices $\textbf{X'}$ and $\textbf{Y'}$. Thus, they are sparse with a few $1$...
0
votes
1answer
16 views
Algebraic transformation — where is my mistake?
I tried to find the estimators of $\hat{\beta_1}$ and $\hat{\beta_0}$ via the least-squares method algebraically. Somehow I seem to have messed up.
Can you tell me where?
My Calculations.
1
vote
1answer
48 views
How to derive the variance of the mean of predictions from a linear regression model?
The context is linear regression analysis for estimating a sample mean. Assume the usual multivariate linear model: $$Y = X\beta + \epsilon$$ with $X$ a $n \times p$ covariate matrix with intercept, $...
0
votes
0answers
38 views
Textbook to learn one-way and two-way ANOVA
I'm looking for a good (detailed, mathematical) textbook to learn one-way and two-way ANOVA.
I'm new to statistics and I was learning linear regression from D. C. Montgomery, E. A. Peck, G. G. Vining,...
0
votes
0answers
18 views
Relationship between different types of correlation coefficients
Let,
$r_{1(2.34...p)}$ = Correlation between $x_1$ and $x_{2.34...p}$. The latter being the residuals after regressing $x_2$ on $x_3 , x_4 ....x_p$.
$r_{1.234..p}$ = Multiple correlation coefficient ...
0
votes
1answer
16 views
Calculating $\hat{\beta_1}$ given $\overline{X}$, $\overline{Y}$, their standard deviations and the correlation coefficient.
In a linear regression problem I'm asked to find $\hat{\beta_1}$ given only the following information:
$\overline{X} = 163.5$
$\overline{Y} = 874.1$
$\sigma_X = 16.2$
$\sigma_Y = 54.2$
$r = -0.774$
...
1
vote
0answers
17 views
Converting back to unnormalized domain after normalizing
I'm trying to perform a linear regression on a dataset however, without normalizing the data I couldn't achieve any results therefore I was forced to normalize it. Now my graph looks fine for the ...
0
votes
0answers
14 views
compare Bayesian linear regression vs standard linear regression
1st question,
I recently learnt bayesian linear regression, but I'm confused that in what situation we should use bayesian linear regression, and when to use standard linear regression? What is the ...
0
votes
1answer
52 views
Help in understanding Matrix Differentiation laws used in Stochastic Gradient Descent
I come from a programming background. I am familiar with scalar calculus but not so much with vector/matrix calculus.
I am trying to understand stochastic gradient descent for multiple linear ...
0
votes
0answers
11 views
how write residual by adding outliers?
I'm reading a paper about outliers and its detection. there they suppose a initial group of observatios and next they add a group of high leverage identical outliers. where to the initial observations ...
1
vote
1answer
35 views
how to bound the hat matrix?
I'm reading a paper about linear regression and in some point they define:
$$ w_{ij}=\frac{h^{2}_{ji}}{ph_{ii}(1-h_{jj})^{2}}$$, where $h_{ij}$ are the elements of the hat matrix.
The problem is ...
0
votes
1answer
39 views
Multivariate Linear regression with fewer trained parameters
I have a multivariate linear regression problem to solve for identifying a dynamic greybox model.
Normally I would formulate it in this form:
Given $n$ number of observations, $m$ response variables,...
2
votes
1answer
62 views
Isotonic regression with a linear constraint
I'm trying to find a direct approach to solving (for some fixed vector $y$):
$$
\begin{aligned}
\min & \; \|x - y \|^2 \\
\mbox{s.t. } & \alpha^\top x \leq 0 \\
& x_i \...
0
votes
1answer
24 views
Calculating error on slope of graph
I'm trying to find the rate of change and the error on that rate based on 7 measurements points and the assumption that the trend is linear. My calculations are below:
$$
\begin{array}{|c|c|c|c|}
\...
0
votes
0answers
9 views
How find the Cook's distance expected value for a large n
I'm trying to prove that for a large n, the Cook expected value is approximated by:
$$E[D_{i}]\approx \frac{h_{ii}}{p(1-h_{ii})}$$. Do you know where can i read the prove of that?. i tried to do it ...
0
votes
0answers
12 views
How do I derive a Least Square Estimator of a linear combination of two variables?
I am working on a problem where I have the following model:
lm(Y ~ x1 x2)
If I have the output of this general model in R, is it possible to derive the LSE of:
<...
0
votes
1answer
31 views
line nearest to a set of points
In stack overflow there are at least two questions asking how to find the straight line in a 2D space nearest to a given set of points $\vec{a}_i$ for $i = 1...N$: question 1 question 2.
In both ...
0
votes
1answer
42 views
what is the -(2/n) in the gradient descent
I am trying to explore how gradient descent works. I found a Formula
$$\begin{aligned} \frac { \partial } { \partial \mathrm { m } } & =
\frac { 2 } { N } \sum _ { i = 1 } ^ { N } - x _ { i }...
