Questions tagged [linear-regression]
For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.
0
votes
0answers
17 views
How do I remove points to subtract the $b$ term in a linear regression fit?
I have a scatterplot of points with a fit of $y = mx + b$.
I would like to only keep the points such that the linear fit would be:
$y = mx$.
How do I subtract out the points contributing to the $b$ ...
1
vote
1answer
35 views
Proof for the asymptotic correlation coefficient $\lim_{x\to\infty} \rho_{{\hat{\beta_0}},{\hat{\beta_1}}} = -\frac{\sqrt{3}}{2}$
Be my regression model:
$$Y_i = 1 + X_i + \epsilon_i, \hspace{5mm} \epsilon_i \sim N(0,1), \hspace{5mm} X_i = \frac{i-1}{n-1}, \;\;\; i = 1, 2, ..., n$$
I want to prove that the asymptotic correlation ...
1
vote
0answers
10 views
Proving almost sure convergence of linear regression coefficients
In the context of simple linear regression, suppose that $\epsilon_i, \ i=1,...,n$ are i.i.d and $ |n^{-1}\sum_{i=1}^{n}x_{i}| \rightarrow |\mu| < \infty$ where n $\rightarrow \infty$ and var(x) = $...
0
votes
0answers
11 views
Restricted Weighted Linear Regression in R
I have to follwing issue. I would like to run a linear regression imposing a constraint on the weighted coefficients. Let me construct an example:
Consider the following cross-sectional regression
$...
0
votes
0answers
19 views
Prove the negative log-likelihood function is a Lipschitz function
Given $n$ data points $\{(z_i, \phi_i)\}_{i=1,\ldots,n}$, $\phi_i \in \mathbb{R}^d$, $z_i \in \mathbb{R}^m$, consider the negative log-likelihood function
$F(X,\Theta) = \left[ \dfrac{1}{n} \sum_{i=1}...
1
vote
1answer
18 views
What parameter is necessary for the AIC criterion applied to linear regression models?
I did a linear regression model (OLS) and a spatial autoregressive model (Spatial lag). I read that for comparing these models I need to use the Akaike information criterion (AIC). The formula is the ...
0
votes
3answers
102 views
Does $A^TA$ always result in a square matrix?
And does the resulting $A^TA$ matrix always have an inverse to solve for $\vec{w}$ in
$$A^TA\vec{w}=A^T\vec{t}$$ ?
0
votes
0answers
12 views
How to scale a weighted variable for linear regression?
I would like to scale a variable such that after weighting it is scaled to mean zero and standard deviation one:
$\sum_{i} w_{i} x_{i} = 0$ and
$\sum_{i} w_{i} x_{i}^2 = 1$
where $w_{i}$ are ...
0
votes
1answer
31 views
Check unity in a regression
I have the following question:
The equation from part c:
My Solution
I'm thinking of solving this problem using the RESET test so something along the lines of:
$$yl_t = -0.027 + 0.537 kl_t + \hat{...
0
votes
1answer
23 views
dimensionality reduction of linear regression model
Context:
My goal is to visualise in a plot (or multiple plots) how my linear plot fits the data.
So, I have a linear system which takes as inputs $\vec{x}=\begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_m \...
1
vote
0answers
19 views
How to check if a regression has a problem of multicollinearity?
I have the following problem:
My Solution:
I'm trying to solve this problem by doing the following:
so we know that:
$$ R^2 = \frac{SSE/(T-K)}{SST/(T-1)}$$
By plotting all the know values we get:
$...
0
votes
1answer
36 views
Why is the total sum of squares equals to explained sum of squares + residual sum of squares?
Given set of sample points $(x_i,y_i)$ in the 2-dimentional space,
Let $(x_i,\hat{y_i})$ be the corresponding points on the regression line. And let $\bar{y}$ be the average of $y$.
I want to know ...
0
votes
0answers
12 views
Solution of an endogeneity model without instrumental variable, where $E[\varepsilon | x]\neq0$
I know that using Instrumental Variable (IV) solves endogeneity, feedback, unbiased problems and other situations.
I was wondering, besides IV, what other method comes to your minds to solve a model ...
0
votes
0answers
14 views
What is standard deviation, variance and R square in Linear Regression?
I am learning regression and classification the past week as a part of my machine learning class. So, I have understood what is linear regression and how the coefficient error values are calculated ...
0
votes
0answers
12 views
Effects of parameter $\lambda$ in ridge regression and cross validation
Problem
Role of $\lambda$
In ridge regression, when (slim) SVD of the design matrix is specified as $\mathbf{X}=\mathbf{UDV}^T \in \mathbb{R}^{n\times p}$, the optimal parameter could be written as
$...
0
votes
0answers
13 views
Functional data analysis: How to find the error term inorder to calculate the variance
Functional data analysis: How to find the error term in order to calculate the variance
I am trying to do functional data analysis problem mentioned below:
What I don't know is how to find the ...
0
votes
0answers
18 views
Conditional Interpretations of Linear Regression
We estimate a linear regressor in the 1 dimensional with x and y random variables with zero mean:
y/x = $\alpha$ x
We can rewrite this using the variance of the variables as:
y/x = $\rho \frac{\...
0
votes
0answers
27 views
Fitting a quadratic model using simple linear regression
Suppose that the true model is
$Y_i = \beta_0 + \beta_1X_i + \beta_2X_i^2 + \epsilon_i, 1 \leq i \leq n,$
where $ \{\epsilon_i\}$ are i.i.d normal random variables with mean zero and variance $ \...
0
votes
1answer
43 views
General Gauss-Markov theorem [closed]
$Y=XB+u$ where $X$ is a non random $n\times k$ Matrix, $\textrm{rank}(X)=k, E(u)=0, E(uu')=\sigma^2\Omega$, How to form $(1)$ How to proof $(2)$ the general Gauss-Markov theorem?
0
votes
0answers
12 views
Inverse of regression coefficient
rearranging the normal linear regression function with one variable yields
$$-b_0/b_1+1/b_1*Y_i-1/b_1*u_i=X_i$$.
The question for me is why the regression coefficient in front of Y, $1/b_1$, is not ...
0
votes
0answers
58 views
Proof in regression model
Im studying for an exam, i don't have the solution, so I hope some of you guys can help me. I have tried a lot but i can't do this proof.
Here is the task:
Suoppose we have the linear regression ...
0
votes
0answers
17 views
Valid multivariate Gaussian model
I wanted to know if someone could explain the reason why if we have $n < (1/2)p(p+1)$ the number of parameters $p$ is greater than the number of samples $n$. I just can't figure out where does this ...
0
votes
0answers
29 views
Optimal solution for linear regression problem
I have an assignment for university about the linear regression problem and the optimal solution.
For the linear regression problem
$\min_w ||Y - X\omega||^2$,
$\omega^*$ is the optimal solution. ...
-2
votes
1answer
18 views
I can't find the product moment coefficient of correlation? [closed]
These are the equations of least square regression lines:
$ Y = 20.8 - 0.219 X $ ($Y$ on $ X$)
$X = 16.2 - 0.785 Y$ ($X$ on $Y$)
Find the coefficient of correlation $r$.
0
votes
0answers
20 views
Variance of Residuals in Multiple Linear Regression
If I have $n$ variables $x_1,\dots,x_d$ in my dataset, and I regress the first one against all the others i.e.
$x_1=a_{12}x_2 + a_{13}x_3 + \dots + a_{1d} x_d + \epsilon_1$,
how can I calculate $\...
0
votes
0answers
6 views
Predictions from bivariate regression model
I have a bivariate (2 responses variables) regression model and I am stuck with the following predictions model and prediction
Basically I need to predict Y_1 taking into account also the correlation ...
1
vote
1answer
101 views
Basic application of category theory to data science
Linear regression is the algorithm that, given a set of vectors ${\bf x}_i \in \mathbb{R}^p$, and a set of targets $y_i \in \mathbb{R}$, returns a vector ${\bf w} \in \mathbb{R}^p$ minimizing the ...
1
vote
0answers
27 views
Linear regression model
Assume the usual linear regression model with $Y = X \beta + \epsilon$, where $X$ is fixed and known and $E(\epsilon) = 0$ and $\operatorname{var}(\epsilon) = \sigma^2I$.
Let $a$ be an $n$-...
0
votes
1answer
29 views
Regression: width of confidence interval vs t-statistic
My question is about confidence intervals of the slopes estimated in a multivariate regression.
I would like to clear up something that is probably a fault somewhere in my understanding. As I ...
1
vote
0answers
22 views
Regression — gradient descent versus statistical methods
In a machine learning course, the professor described the gradient descent method for calculating the regression line. Shortly, we're looking for the $a$ and $b$ in $y=ax+b$ which describes the line. ...
0
votes
0answers
22 views
Error Term, Regression
Assume I have $Y_i=A+BX_i + \epsilon_i$
For a multiple linear regression I have,
$Y_i=A+B_1X_1+B_2X_2+....+B_NX_N+\epsilon$
The question is, what is the distribution of $\epsilon$, knowing that $\...
1
vote
3answers
44 views
What method does a calculator use to calculate a linear regression line?
Take three coordinates $(1,1)$, $(3,2)$ and $(4,3)$.
My calculator returns the linear regression line: $$y=0.6429x+0.2857$$ of the form $$y = ax +b$$ correct to four significant figures for constants ...
2
votes
1answer
27 views
Time-series modeling
I'm wondering what methods can be used to predict a future value using past values. I looked into linear regression modeling, but this doesn't allow for a time value.
As an example, say I have an ...
0
votes
0answers
28 views
Proof for normal posterior distribution for a data for linear regression.
Let $D=(x_1,x_2,....X_N)$ be the data and $x_i$ is D-dimensional data. For linear regression the likelihood is as follows
$$p(y|X,w, u, \sigma^2) = N(y|u + Xw, \sigma^2 I_N) \propto exp(-\frac{1}{2\...
0
votes
0answers
25 views
Interpret model estimates after log transformation
I sat up a mixed-effects linear model with the dependent variable log-transformed (in oder to get it normal distributed and as ist is common with this kind of data in other publications). All fixed ...
0
votes
0answers
21 views
find $\operatorname{Var}(\hat\epsilon_i)=\sigma^2({{n-1}\over n}-{{(x_i-\bar x)^2}\over{\sum_{i=1}^n}(x_i-\bar x)^2})$
I'm studying question 16.19 in the book Probability Essentials. The question is to show that: $\operatorname{Var}(\hat\epsilon_i)=\sigma^2({{n-1}\over n}-{{(x_i-\bar x)^2}\over{\sum_{i=1}^n}(x_i-\bar ...
0
votes
1answer
547 views
Correlation between Least Square Parameters obtained in Linear Regression
I am trying to solve Q 22 below:
Upon minimizing the squared errors I get the following values of the parameters:
$$\hat{\alpha} = \overline{y}$$
$$\hat{\beta} = \frac{\sum x_i y_i - \overline{x}\...
2
votes
1answer
52 views
Rank of sub-matrix of projection matrix
Consider the projection matrix in Linear Regression $P=X(X^TX)^{-1}X^T$. If we have $n$ points, $P$ is an $n$ x $n$ matrix. We also know it satisfies a number of properties, including that it's ...
1
vote
0answers
8 views
Regression task on desirable subsets with limited supervision
Suppose I have a set of n elements. I want to have a model for how "desirable" a subset of those elements is when evaluated by a person. The training/testing data is a number of given subsets, each ...
1
vote
1answer
20 views
How to interpret the vectors and design matrix in a linear model
In regression, linear models are of the form:
$$y_i = \pmb z_i^T \pmb\beta_i + \epsilon_i$$
Or we can write this in a more general form with vectors and a design matrix:
$$\pmb y = \pmb Z \pmb \beta ...
0
votes
1answer
39 views
What is weighted and unweighted linear regression in machine learning?
I'm taking Stanford's CS229 ML course and while studying about "parametric algorithms", Prof. Andrew Ng says that this class of algorithms has a fix number of parameters (parameters are also called as ...
2
votes
1answer
23 views
Bias and variance of IV estimation
NOTE: I have asked this question on stat.stackoverflow but got no answers/comments. Hence I decide to ask it on the math.stackoverflow platform as well.
I'm studying IV estimation by myself and have ...
0
votes
0answers
31 views
what are $SSE$ and $S_{y,x}$?
I'm a BC student who is trying to solve some statistic quizzes.
there is a multiple choice question that is this:
In a simple regression model $y=a+bx+e$ our given data is this:
$$ \bar{x} = 2, \...
0
votes
1answer
47 views
$Y_1,Y_2,Y_3$ are uncorrelated rvs such that $E(Y_1)=\beta_1+\beta_2$,$E(Y_2)=2\beta_1$ and $E(Y_3)=\beta_1-\beta_2$
Let $Y_1,Y_2,Y_3$ be uncorrelated random variables with common variance $\sigma^2 > 0 $ such that $E(Y_1)=\beta_1+\beta_2$,$E(Y_2)=2\beta_1$ and $E(Y_3)=\beta_1-\beta_2$
where $\beta_1$ and $\...
0
votes
2answers
28 views
Including the (0,0) point in regression
I have run a simple linear regression in Rstudio with two variables and got the following relation:
y = 30000+1.95x
Which is reasonably fair. My only concern is that, practically the (0,0) point ...
1
vote
1answer
27 views
Logistic Regression Explanation
I have two questions regarding logistic regression.
1) I understand that the results of a logistic regression model yield a table stating coefficients together with a p-statistic for each variable . ...
0
votes
2answers
26 views
Linear Regression Assumption
the following website states that linear regression assumes a linear relationship between dependent and independent variables:
"First, linear regression needs the relationship between the independent ...
0
votes
1answer
36 views
Linear Regression Assumption: Normality of residual vs normality of variables
I have read in many places, including stack exchange, that in order to carry linear regression analysis the residuals have to be normal. This is required because most of the statistical results, ...
0
votes
0answers
17 views
Operating with penalized regression models
Assume we are working with a penalized linnear regression model. We have the following optimization problem:
\begin{equation}
\min_{\beta}\left\{\left\lVert y-X\beta\right\rVert_2^2+\lambda\sum_{j=1}^...
0
votes
1answer
27 views
Proof of Batch Gradient Descent's cost function gradient vector
In the book Hands-On Machine Learning with Scikit-Learn & TensorFlow, the author only showed the formula for the Batch Gradient Descent method, such as:
$ \dfrac{\partial}{\partial \theta_{j}} ...
