Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [linear-regression]

For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.

0
votes
0answers
17 views

How do I remove points to subtract the $b$ term in a linear regression fit?

I have a scatterplot of points with a fit of $y = mx + b$. I would like to only keep the points such that the linear fit would be: $y = mx$. How do I subtract out the points contributing to the $b$ ...
1
vote
1answer
35 views

Proof for the asymptotic correlation coefficient $\lim_{x\to\infty} \rho_{{\hat{\beta_0}},{\hat{\beta_1}}} = -\frac{\sqrt{3}}{2}$

Be my regression model: $$Y_i = 1 + X_i + \epsilon_i, \hspace{5mm} \epsilon_i \sim N(0,1), \hspace{5mm} X_i = \frac{i-1}{n-1}, \;\;\; i = 1, 2, ..., n$$ I want to prove that the asymptotic correlation ...
1
vote
0answers
10 views

Proving almost sure convergence of linear regression coefficients

In the context of simple linear regression, suppose that $\epsilon_i, \ i=1,...,n$ are i.i.d and $ |n^{-1}\sum_{i=1}^{n}x_{i}| \rightarrow |\mu| < \infty$ where n $\rightarrow \infty$ and var(x) = $...
0
votes
0answers
11 views

Restricted Weighted Linear Regression in R

I have to follwing issue. I would like to run a linear regression imposing a constraint on the weighted coefficients. Let me construct an example: Consider the following cross-sectional regression $...
0
votes
0answers
19 views

Prove the negative log-likelihood function is a Lipschitz function

Given $n$ data points $\{(z_i, \phi_i)\}_{i=1,\ldots,n}$, $\phi_i \in \mathbb{R}^d$, $z_i \in \mathbb{R}^m$, consider the negative log-likelihood function $F(X,\Theta) = \left[ \dfrac{1}{n} \sum_{i=1}...
1
vote
1answer
18 views

What parameter is necessary for the AIC criterion applied to linear regression models?

I did a linear regression model (OLS) and a spatial autoregressive model (Spatial lag). I read that for comparing these models I need to use the Akaike information criterion (AIC). The formula is the ...
0
votes
3answers
102 views

Does $A^TA$ always result in a square matrix?

And does the resulting $A^TA$ matrix always have an inverse to solve for $\vec{w}$ in $$A^TA\vec{w}=A^T\vec{t}$$ ?
0
votes
0answers
12 views

How to scale a weighted variable for linear regression?

I would like to scale a variable such that after weighting it is scaled to mean zero and standard deviation one: $\sum_{i} w_{i} x_{i} = 0$ and $\sum_{i} w_{i} x_{i}^2 = 1$ where $w_{i}$ are ...
0
votes
1answer
31 views

Check unity in a regression

I have the following question: The equation from part c: My Solution I'm thinking of solving this problem using the RESET test so something along the lines of: $$yl_t = -0.027 + 0.537 kl_t + \hat{...
0
votes
1answer
23 views

dimensionality reduction of linear regression model

Context: My goal is to visualise in a plot (or multiple plots) how my linear plot fits the data. So, I have a linear system which takes as inputs $\vec{x}=\begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_m \...
1
vote
0answers
19 views

How to check if a regression has a problem of multicollinearity?

I have the following problem: My Solution: I'm trying to solve this problem by doing the following: so we know that: $$ R^2 = \frac{SSE/(T-K)}{SST/(T-1)}$$ By plotting all the know values we get: $...
0
votes
1answer
36 views

Why is the total sum of squares equals to explained sum of squares + residual sum of squares?

Given set of sample points $(x_i,y_i)$ in the 2-dimentional space, Let $(x_i,\hat{y_i})$ be the corresponding points on the regression line. And let $\bar{y}$ be the average of $y$. I want to know ...
0
votes
0answers
12 views

Solution of an endogeneity model without instrumental variable, where $E[\varepsilon | x]\neq0$

I know that using Instrumental Variable (IV) solves endogeneity, feedback, unbiased problems and other situations. I was wondering, besides IV, what other method comes to your minds to solve a model ...
0
votes
0answers
14 views

What is standard deviation, variance and R square in Linear Regression?

I am learning regression and classification the past week as a part of my machine learning class. So, I have understood what is linear regression and how the coefficient error values are calculated ...
0
votes
0answers
12 views

Effects of parameter $\lambda$ in ridge regression and cross validation

Problem Role of $\lambda$ In ridge regression, when (slim) SVD of the design matrix is specified as $\mathbf{X}=\mathbf{UDV}^T \in \mathbb{R}^{n\times p}$, the optimal parameter could be written as $...
0
votes
0answers
13 views

Functional data analysis: How to find the error term inorder to calculate the variance

Functional data analysis: How to find the error term in order to calculate the variance I am trying to do functional data analysis problem mentioned below: What I don't know is how to find the ...
0
votes
0answers
18 views

Conditional Interpretations of Linear Regression

We estimate a linear regressor in the 1 dimensional with x and y random variables with zero mean: y/x = $\alpha$ x We can rewrite this using the variance of the variables as: y/x = $\rho \frac{\...
0
votes
0answers
27 views

Fitting a quadratic model using simple linear regression

Suppose that the true model is $Y_i = \beta_0 + \beta_1X_i + \beta_2X_i^2 + \epsilon_i, 1 \leq i \leq n,$ where $ \{\epsilon_i\}$ are i.i.d normal random variables with mean zero and variance $ \...
0
votes
1answer
43 views

General Gauss-Markov theorem [closed]

$Y=XB+u$ where $X$ is a non random $n\times k$ Matrix, $\textrm{rank}(X)=k, E(u)=0, E(uu')=\sigma^2\Omega$, How to form $(1)$ How to proof $(2)$ the general Gauss-Markov theorem?
0
votes
0answers
12 views

Inverse of regression coefficient

rearranging the normal linear regression function with one variable yields $$-b_0/b_1+1/b_1*Y_i-1/b_1*u_i=X_i$$. The question for me is why the regression coefficient in front of Y, $1/b_1$, is not ...
0
votes
0answers
58 views

Proof in regression model

Im studying for an exam, i don't have the solution, so I hope some of you guys can help me. I have tried a lot but i can't do this proof. Here is the task: Suoppose we have the linear regression ...
0
votes
0answers
17 views

Valid multivariate Gaussian model

I wanted to know if someone could explain the reason why if we have $n < (1/2)p(p+1)$ the number of parameters $p$ is greater than the number of samples $n$. I just can't figure out where does this ...
0
votes
0answers
29 views

Optimal solution for linear regression problem

I have an assignment for university about the linear regression problem and the optimal solution. For the linear regression problem $\min_w ||Y - X\omega||^2$, $\omega^*$ is the optimal solution. ...
-2
votes
1answer
18 views

I can't find the product moment coefficient of correlation? [closed]

These are the equations of least square regression lines: $ Y = 20.8 - 0.219 X $ ($Y$ on $ X$) $X = 16.2 - 0.785 Y$ ($X$ on $Y$) Find the coefficient of correlation $r$.
0
votes
0answers
20 views

Variance of Residuals in Multiple Linear Regression

If I have $n$ variables $x_1,\dots,x_d$ in my dataset, and I regress the first one against all the others i.e. $x_1=a_{12}x_2 + a_{13}x_3 + \dots + a_{1d} x_d + \epsilon_1$, how can I calculate $\...
0
votes
0answers
6 views

Predictions from bivariate regression model

I have a bivariate (2 responses variables) regression model and I am stuck with the following predictions model and prediction Basically I need to predict Y_1 taking into account also the correlation ...
1
vote
1answer
101 views

Basic application of category theory to data science

Linear regression is the algorithm that, given a set of vectors ${\bf x}_i \in \mathbb{R}^p$, and a set of targets $y_i \in \mathbb{R}$, returns a vector ${\bf w} \in \mathbb{R}^p$ minimizing the ...
1
vote
0answers
27 views

Linear regression model

Assume the usual linear regression model with $Y = X \beta + \epsilon$, where $X$ is fixed and known and $E(\epsilon) = 0$ and $\operatorname{var}(\epsilon) = \sigma^2I$. Let $a$ be an $n$-...
0
votes
1answer
29 views

Regression: width of confidence interval vs t-statistic

My question is about confidence intervals of the slopes estimated in a multivariate regression. I would like to clear up something that is probably a fault somewhere in my understanding. As I ...
1
vote
0answers
22 views

Regression — gradient descent versus statistical methods

In a machine learning course, the professor described the gradient descent method for calculating the regression line. Shortly, we're looking for the $a$ and $b$ in $y=ax+b$ which describes the line. ...
0
votes
0answers
22 views

Error Term, Regression

Assume I have $Y_i=A+BX_i + \epsilon_i$ For a multiple linear regression I have, $Y_i=A+B_1X_1+B_2X_2+....+B_NX_N+\epsilon$ The question is, what is the distribution of $\epsilon$, knowing that $\...
1
vote
3answers
44 views

What method does a calculator use to calculate a linear regression line?

Take three coordinates $(1,1)$, $(3,2)$ and $(4,3)$. My calculator returns the linear regression line: $$y=0.6429x+0.2857$$ of the form $$y = ax +b$$ correct to four significant figures for constants ...
2
votes
1answer
27 views

Time-series modeling

I'm wondering what methods can be used to predict a future value using past values. I looked into linear regression modeling, but this doesn't allow for a time value. As an example, say I have an ...
0
votes
0answers
28 views

Proof for normal posterior distribution for a data for linear regression.

Let $D=(x_1,x_2,....X_N)$ be the data and $x_i$ is D-dimensional data. For linear regression the likelihood is as follows $$p(y|X,w, u, \sigma^2) = N(y|u + Xw, \sigma^2 I_N) \propto exp(-\frac{1}{2\...
0
votes
0answers
25 views

Interpret model estimates after log transformation

I sat up a mixed-effects linear model with the dependent variable log-transformed (in oder to get it normal distributed and as ist is common with this kind of data in other publications). All fixed ...
0
votes
0answers
21 views

find $\operatorname{Var}(\hat\epsilon_i)=\sigma^2({{n-1}\over n}-{{(x_i-\bar x)^2}\over{\sum_{i=1}^n}(x_i-\bar x)^2})$

I'm studying question 16.19 in the book Probability Essentials. The question is to show that: $\operatorname{Var}(\hat\epsilon_i)=\sigma^2({{n-1}\over n}-{{(x_i-\bar x)^2}\over{\sum_{i=1}^n}(x_i-\bar ...
0
votes
1answer
547 views

Correlation between Least Square Parameters obtained in Linear Regression

I am trying to solve Q 22 below: Upon minimizing the squared errors I get the following values of the parameters: $$\hat{\alpha} = \overline{y}$$ $$\hat{\beta} = \frac{\sum x_i y_i - \overline{x}\...
2
votes
1answer
52 views

Rank of sub-matrix of projection matrix

Consider the projection matrix in Linear Regression $P=X(X^TX)^{-1}X^T$. If we have $n$ points, $P$ is an $n$ x $n$ matrix. We also know it satisfies a number of properties, including that it's ...
1
vote
0answers
8 views

Regression task on desirable subsets with limited supervision

Suppose I have a set of n elements. I want to have a model for how "desirable" a subset of those elements is when evaluated by a person. The training/testing data is a number of given subsets, each ...
1
vote
1answer
20 views

How to interpret the vectors and design matrix in a linear model

In regression, linear models are of the form: $$y_i = \pmb z_i^T \pmb\beta_i + \epsilon_i$$ Or we can write this in a more general form with vectors and a design matrix: $$\pmb y = \pmb Z \pmb \beta ...
0
votes
1answer
39 views

What is weighted and unweighted linear regression in machine learning?

I'm taking Stanford's CS229 ML course and while studying about "parametric algorithms", Prof. Andrew Ng says that this class of algorithms has a fix number of parameters (parameters are also called as ...
2
votes
1answer
23 views

Bias and variance of IV estimation

NOTE: I have asked this question on stat.stackoverflow but got no answers/comments. Hence I decide to ask it on the math.stackoverflow platform as well. I'm studying IV estimation by myself and have ...
0
votes
0answers
31 views

what are $SSE$ and $S_{y,x}$?

I'm a BC student who is trying to solve some statistic quizzes. there is a multiple choice question that is this: In a simple regression model $y=a+bx+e$ our given data is this: $$ \bar{x} = 2, \...
0
votes
1answer
47 views

$Y_1,Y_2,Y_3$ are uncorrelated rvs such that $E(Y_1)=\beta_1+\beta_2$,$E(Y_2)=2\beta_1$ and $E(Y_3)=\beta_1-\beta_2$

Let $Y_1,Y_2,Y_3$ be uncorrelated random variables with common variance $\sigma^2 > 0 $ such that $E(Y_1)=\beta_1+\beta_2$,$E(Y_2)=2\beta_1$ and $E(Y_3)=\beta_1-\beta_2$ where $\beta_1$ and $\...
0
votes
2answers
28 views

Including the (0,0) point in regression

I have run a simple linear regression in Rstudio with two variables and got the following relation: y = 30000+1.95x Which is reasonably fair. My only concern is that, practically the (0,0) point ...
1
vote
1answer
27 views

Logistic Regression Explanation

I have two questions regarding logistic regression. 1) I understand that the results of a logistic regression model yield a table stating coefficients together with a p-statistic for each variable . ...
0
votes
2answers
26 views

Linear Regression Assumption

the following website states that linear regression assumes a linear relationship between dependent and independent variables: "First, linear regression needs the relationship between the independent ...
0
votes
1answer
36 views

Linear Regression Assumption: Normality of residual vs normality of variables

I have read in many places, including stack exchange, that in order to carry linear regression analysis the residuals have to be normal. This is required because most of the statistical results, ...
0
votes
0answers
17 views

Operating with penalized regression models

Assume we are working with a penalized linnear regression model. We have the following optimization problem: \begin{equation} \min_{\beta}\left\{\left\lVert y-X\beta\right\rVert_2^2+\lambda\sum_{j=1}^...
0
votes
1answer
27 views

Proof of Batch Gradient Descent's cost function gradient vector

In the book Hands-On Machine Learning with Scikit-Learn & TensorFlow, the author only showed the formula for the Batch Gradient Descent method, such as: $ \dfrac{\partial}{\partial \theta_{j}} ...