Questions tagged [linear-regression]

For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.

2
votes
2answers
37 views

how to find the distribution of the least square estimator $\beta$?

i'm solving a problem that involve a linear model, and i'm trying to get the distribution of the least square estimator $\beta$. i found in a book that: $\widehat{\beta}\sim N_{p}(\beta, (X^{\...
0
votes
0answers
16 views

find a confidence interval in a linear model problem

i'm trying to solve a problem that involve a linear model given its normal equations, and the errors have a normal distribution but i'm a little lost. the problem is about construct a 95% ...
0
votes
0answers
29 views

Integration of supremum of Gaussian pdf

I am trying to compute certain properties of a Linear Regression model with Gaussian Noise. Under a series of hypothesis I assumed that my parameter $\beta : y = \beta x + (\epsilon\sim \mathcal{N}(0,\...
1
vote
1answer
46 views

estimate of a slope in the simple linear regression model $y=\beta_0+\beta_1 x+\epsilon$

I have two formulas for estimate of a slope in the simple linear regression model $y=\beta_0+\beta_1 x+\epsilon$: $\hat{\beta_1}=\frac{\sum_{i=1}^N(x_i-\bar{x})(y_i-\bar{y})}{\sum^N_{i=1}(x_i-\bar{x}...
0
votes
0answers
21 views

Are most resources misleading about the exact Euclidian distance that are being minimized in linear least squares?

Most illustrations indicate that the distance of interest in the sum of the squares is the perpendicular distance between individual data points and the hyperplane defined by the linear model. The ...
1
vote
1answer
22 views

Expectations and Variance of linear regression model

Consider, Y = c + βX + ε where E(ε|X) = 0 and Var(ε|X) = (σ sub ε)^2. Assume Var(X) = (σ sub X)^2. Find Var(E(Y|X)). So far I have, (E(Y|X)) = E(E(Y|X)^2) - (E(E(Y|X)))^2 I'm not sure where to ...
0
votes
1answer
131 views

deriving the formula for multinomial linear regression [closed]

I am trying to understand why $\theta_{MLE} = (X^TX)^{-1}X^Ty$ for multinomial linear regression in which we have the Frobenius norm for $min||y-X\theta||^2$ Looking at this tutorial, I have hard time ...
0
votes
0answers
78 views

Fitting a curve to $N$ dimensional data

I have a data set with $N$ independent variables and one dependent variable(function of all the $N$ independent variables). The dependent variable is either $0$ or $1$ (like a step function). I am ...
0
votes
0answers
45 views

Weighted least squares estimator for a non-zero intercept regression

If I have the following regression model with intercept $\alpha$ $$y=X\beta + \alpha + \epsilon$$ Is the Weighted Least Squares (WLS) estimator for $\beta$ the same as in the zero-intercept ...
0
votes
1answer
21 views

Trying to understand errors-in-variables and how this affect the choice of number of subjects in a study

I am twisting my brain on some voluntary exercises we have received in our Data Analytics class in my study. We have a dataset of respondents to a imaginary analysis, and by using linear regression to ...
3
votes
1answer
255 views

What's the variance of intercept estimator in multiple linear regression?

Suppose a linear regression model $Y=Xβ+ε$ where $X$ is an $n$-by-$(k+1)$ matrix and $\epsilon$ follows $N(0,\sigma^2I_n)$. $k$ is the number of explanatory variables. The first column of $X$ is one (...
0
votes
0answers
10 views

Decrease Beta bewteen to variables

I have two time series of return data. One is global equity and one is a seperate stock. I would like to know if there is a theoretical way of decreasing the EQ Beta of the stock? To be more clear: ...
0
votes
0answers
28 views

How to solve multivariate linear regression?

Suppose $x \in R^2$, $A\in R^{2X2}$ and $ b\in R^2$. $\hat{y}=Ax+b$ where $\hat{y}$ is predicted output. We have to find $A$ and $b$ in terms of $X$ and $Y$ such that squared L2 distance between $\hat{...
1
vote
1answer
160 views

symmetry of Regression line when SDx equals SDy and non intutive results

This question is taken from Freedman In a certain class, midterm scores average out to $60$ with an SD of $15,$ as do scores on the final. The correlation between midterm scores and final ...
0
votes
0answers
34 views

Correlation coefficient in terms of standard units: intuition

Correlation coefficient $$r = \frac{1}{n}\sum_{i=1}^n\frac{(x_i-\bar x)(y_i-\bar y)}{\sigma_x\cdot\sigma_y}$$ But for a given data point $x_i$ and predicted value $y_p$, $$\frac{(y_p-\bar y)}{\...
0
votes
2answers
46 views

Proof for Simple Linear Regression: What am I doing wrong?

I am trying to prove the well known formula for simple linear regression $$SS_{TOTAL}=SS_{MODEL}+SS_{ERROR}$$ i.e $$\sum_{i=1}^n (y_i - \bar{y})^2 =\sum_{i=1}^n (\hat{y}_i-\bar{y})^2+ \sum_{i=1}^n(\...
4
votes
2answers
98 views

Correlation coefficient and regression line : Geometric intuition

correlation coefficient $$r = \frac{1}{n}\sum_{i=1}^n\frac{(x_i-\bar x)(y_i-\bar y)}{\sigma_x\cdot\sigma_y}$$ may be thought of as cosine of angle between two $n$-dimensional vectors $$ (x_1- \bar ...
1
vote
1answer
47 views

Coefficient of Determination and Correlation between observed and fitted value in Multiple Linear Regression.

Consider Multiple Linear Model $$y= X\beta + \epsilon$$ Then using Ordinary Least Square, we get estimate of $\beta$ as $$\hat{\beta} = (X^{'}X)^{-1}X^{'}y$$ And $$\hat{y} = X\hat{\beta}$$ $$SS_{Res}= ...
3
votes
1answer
471 views

Basic exponential regression

Background: I've been struggling with an exponential regression problem for about 8 months now (on and off): Vertically translated depreciation curve: Update the exponential regression coefficient ...
0
votes
0answers
14 views

Solve an equation for a set of values within summation operator.

I need to find a set of values, $\eta_k$, in the equations: $$G'(\omega_j) = \sum_{k=1}^9\frac{\eta_k\lambda_k\omega_j^2}{1-(\lambda_k\omega_j)^2} $$ $$G''(\omega_j) = \sum_{k=1}^9\frac{\eta_k\...
3
votes
2answers
108 views

Ridge regression to minimize RMSE instead of MSE

Given a metrix $X$ and a vector $\vec{y}$, ordinary least squares (OLS) regression tries to find $\vec{c}$ such that $\left\| X \vec{c} - \vec{y} \right\|_2^2$ is minimal. (If we assume that $\left\| ...
0
votes
0answers
54 views

Non-recursive Algorithm for Solving Systems of Linear Equations

I'm looking to duplicate Excel's LINEST() functionality using DAX. To do this, I plan to use an ordinary least squares approach which means I need to be able to ...
0
votes
0answers
18 views

relationship between hypothesis test for slope of simple linear regression and r^2

As I understand it, when you fail to reject the null hypothesis that slope=0 for simple linear regression using a t-test - it would suggest that the linear model degenerates into an intercept only ...
1
vote
1answer
54 views

Linear Regression, comparing two models

I have a model that links the heating of a component compared to the frequency applied to it. The model is $\text{Heat} = \alpha \cdot \text{Frequency} + \beta$. Based on the data, I found $\hat{\...
1
vote
1answer
44 views

How to obtain expressions for coefficients from OLS formula?

Consider the standard linear regression model: $y_i = \alpha + \beta D_i + e_i$ where the coefficients are defined by linear projections and $D_i$ is a dummy variable. In the population, the ...
0
votes
1answer
65 views

Are there any properties for matrix A (A= USVt) in Singular Value Decomposition?

I have seen a lot of explanations about SVD with examples figure how it works. like for example figure in this link. by looking the figure it seems that m must be bigger than n (m>n) for matrix A with ...
1
vote
1answer
167 views

Computational Challenge of Multivariate Integration in Sympy

I am creating a piecewise linear approximation for the following equation: $$W = \frac{\theta \left(k \rho\right)^{k}}{2 k^{2}k! \rho \left( 1- \rho\right)^{2} \left(\frac{\left(k \rho\right)^{k}}{\...
0
votes
1answer
31 views

Beta parameter estimation in least squares method by partial derivative

I was reading a particular document on linear regression and I just can't understand how the guy got to the estimation of the beta parameter. His results are as follows: $SQ(\alpha, \beta)=\sum\...
0
votes
0answers
55 views

Convert a matrices multiplications to Ax=b

I have a training data matrix $S_{\tau \times n}$ and actual output $y_{1\times P}$. The weighted parameters for a linear model that maps the input to the output is $$ y = \alpha_{1 \times \tau}S_{\...
1
vote
1answer
62 views

Regressions in case of non-normality

Our variables and residuals are not normally distributed. What we found is that regressions are usually quite robust against violations of normality. But we don't know to which degree, because our ...
0
votes
1answer
181 views

Sparse group lasso derivation of soft thresholding operator via subgradient equations

In the sparse group lasso paper SGLpaper, they derive a condition for $\beta=0$ from the equation: $$ \frac{1}n {X^{(k)}}^T(y-\sum_{l=1}^m X^{(l)} \beta^{(l)}) = (1-\alpha)\lambda u+ \alpha \lambda v$$...
2
votes
1answer
109 views

how fit a model with data following asymptotic / sigmoid pattern

I'm trying to fit data. I assume that the association between dependent and indepdent variable is of the form $$T(y)=aR(x)+b$$ I also know that my data are ressemble either an asymptotic function ...
0
votes
1answer
55 views

How to minimize a function following an asymptotic pattern?

When running simulation of data following a linear function (with noise) with python, I can find that the linear model gives the best fit according to the standard error of the regression: ...
0
votes
0answers
76 views

Show that estimator is the best linear estimator with smallest MSE

Let's consider we have OLS model $Y=X \beta+\epsilon$ , where $E(\epsilon) = 0 $ and $Var(\epsilon)=\sigma^2 I$. Here $\hat{\beta}$ is the least squares estimator of the parameter $\beta$ . Let $z \...
2
votes
2answers
55 views

Learn Noise / Error in Least Squares If We Know Its Form?

Let's say I am doing linear regression and I have a data matrix $A$. And, I know the noise $e_i$, is zero mean (and perhaps we know the distribution, too): $$y_i = a_i^T x + e_i$$ Obviously from ...
0
votes
0answers
253 views

Best fit plane using Least Squares Method and mean error

I'm currently using Least Squares Method in order to calculate Best Fit Plane for a given 3 column vectors [X, Y, Z]. I've noticed that consistently I get that the average of the residuals of Z ...
0
votes
1answer
176 views

Show that $\hat{\beta} $ and $\hat{\sigma^2}$ are unbiased in special case of linear regression model [closed]

Let's consider we have OLS model $Y= X\beta+ \epsilon$, where rows of matrix $X$ are multivariate normal independent vectors with expected value $0$ and variance $\Sigma$. Vector $\epsilon$ is ...
0
votes
0answers
52 views

Is there a hat matrix for gradient descent?

Assume gradient descent on a linear cost function converges for some sequence of examples, is it possible to define a hat matrix for gradient descent that accounts for the gradient update of these ...
0
votes
1answer
25 views

Finding the Variance of a predicted exponentiated linear value

Let me give you some background on my issue. Let's say I have a simple one covariate linear regressor as follows: $$\log\alpha =\alpha_0+\alpha_1x_{i1}.$$ Obviously, to find $\alpha$, we use ...
0
votes
0answers
16 views

Is it possible to perform the regression algorithm on two dependent variables?

I'm trying to change the 3d coordinates of one reference frame 'a' to other reference frame 'b'. For few points in 'a' i'm having corresponding points in 'b'. So based on this, i want to convert all ...
1
vote
0answers
22 views

Formula For Centered Coefficient of Determination

I want to show that in the inhomogeneous regression model $y=\alpha+X\beta+u $ $$ R_*^2=max_{z\in \text{col}(X)}r_{x,y}^2, $$ where $r_{x,y}^2:=\frac{S_{xy}^2}{S_xS_y}$ is the squared empirical ...
1
vote
1answer
85 views

Simplify $ a = \frac{N\sum(xy)-\sum(x)\sum(y)}{N\sum(x^2)-(\sum(x))^2} $ to yield $ a = \frac{\bar{xy} - \bar{x}\bar{y}}{\bar{x^2}-\bar{x}^2}$

I'm working out some multivariable linear regression equations on paper for a class I'm taking, and I'm getting an erroneous factor of N in my solution according to the class. I'm sure it is my error ...
3
votes
1answer
77 views

$\hat{Y} = X^T\hat{\beta}$ Matrix Dimension For Linear Regression Coefficients $\beta$

While reading about least squares implementation for machine learning I came across this passage in the following two photos: Perhaps I’m misinterpreting the meaning of $ \beta $ but if $ X^T$ has ...
0
votes
0answers
51 views

formulation of least squares problems

In general, to use the method of least squares, a linear stochastic system is modeled as: \begin{equation} y = ax + \eta \end{equation} where, $y$, is an observed variable, $x$ is an input while $\...
0
votes
0answers
31 views

Error of the intersection of two linear functions

I have the following linear fits of two data sets $L_1: y=(a_{1}\pm e_{1})x + (b_{1} \pm f_{1})$ $L_2: y=(a_{2}\pm e_{2})x + (b_{2} \pm f_{2})$ How do I calculate the intersection of $L_1$ and $L_2$...
4
votes
3answers
84 views

Why isn't the linear regression coefficient not just the average vector to data points?

I am having trouble intuitively understanding the correctness of the formula to compute the coefficient for the regression line in a linear regression. I know the formula is $$\frac{\sum_{i=1}^N (...
2
votes
4answers
298 views

Does least squares (approximate solution) minimize the orthogonal distance of $b$ to $Ax$, or does it minimize the error projected along the $b$ axis?

I have always been confused about whether the approximate solution to $Ax=b$ is equivalent to minimizing the average distance of all of the $b$ vectors to $Ax$, or whether it is minimizing the ...
0
votes
1answer
35 views

When does the orthogonal fitted value become itself?

Let $m \times a$ full-column matrix $B$, and define the projection matrix onto its column space as $P(B)=B(B'B)^{-1}B'$ which is $m\times m$ matrix. Consider a rectangular matrix $A$, where $A$ is $m ...
1
vote
1answer
34 views

Linear regression property

Suppose that I have the following model: $y_i = \beta_0 + \beta_1 x_{1i} + \beta_2 x_{2i} + u_i$ where $\hat{\beta_k}$, k=0,1,2 , are estimated by the method of least squares, using a sample of ...
0
votes
1answer
49 views

Simplifying $\beta_1$ estimate for a simple linear regression model

For a simple linear regression model I am able to derive the normal equations and solve these to to obtain the following- $$\beta_0=\bar{Y}-\beta_1\bar{X}$$ $$\beta_1=\frac{\sum(X_i-\bar{X})(Y_i-\bar{...