Questions tagged [linear-regression]

For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.

Filter by
Sorted by
Tagged with
9
votes
2answers
12k views

Proof that trace of 'hat' matrix in linear regression is rank of X

I understand that the trace of the projection matrix (also known as the "hat" matrix) X*Inv(X'X)*X' in linear regression is equal to the rank of X. How can we prove that from first principles, i.e. ...
6
votes
3answers
779 views

Why does $A^TAx = A^Tb$ have infinitely many solution algebraically when $A$ has dependent columns?

This is a problem from least square approximation, where we solve the equation $A^TAx = A^Tb$ when $Ax = b$ is unsolvable. The case I am dealing with is when A has dependent columns, i.e. A is an m by ...
6
votes
0answers
75 views

Prime number intercept

Suppose I arrange my (infinite) list of prime numbers in the following way: \begin{array}{c|c}x_i&2&5&11&17&23&31&\cdots\\\hline y_i&3&7&13&19&29&37&...
5
votes
1answer
7k views

what is the variance of a constant matrix times a random vector?

$\newcommand{\Var}{\operatorname{Var}}$In this video is claimed that if the equation of errors in OLS is given by: $$u=y - X\beta$$ Then in the presence of heteroscedasticity the variance of $u$, will ...
5
votes
2answers
6k views

Best Fit Line with 3d Points

Okay, I need to develop an alorithm to take a collection of 3d points with x,y,and z components and find a line of best fit. I found a commonly referenced item from Geometric Tools but there doesn't ...
5
votes
1answer
563 views

Using linear algebra to study number theory?

I've posted a paper on arXiv that outlines a linear algebra approach to number theory. Specifically, I have the following questions: Is it possible to draw connections between the factorization ...
5
votes
0answers
51 views

Can there be a trigonometric function reaching any finite number of points in $\mathbb{R} ^2$

Today in math class we hade a discussion about linear regression, which is all about finding the best (though not perfect) linear equation that passes through a countable set of points, and people ...
5
votes
0answers
279 views

Perturbation theory for least squares for very different A, b

Consider the least squares problem $f(x;A,b) = \|Ax-b\|_2^2$ and define $x^*$ the minimizer of $f(x;\hat A,\hat b)$, and $\hat x$ the minimizer of $f(x; A_2, b_2)$. I want to put some bound on $\|...
4
votes
3answers
85 views

Why isn't the linear regression coefficient not just the average vector to data points?

I am having trouble intuitively understanding the correctness of the formula to compute the coefficient for the regression line in a linear regression. I know the formula is $$\frac{\sum_{i=1}^N (...
4
votes
3answers
5k views

Proof of Gauss-Markov theorem

Theorem: Let $Y=X\beta+\varepsilon$ where $$Y\in\mathcal M_{n\times 1}(\mathbb R),$$ $$X\in \mathcal M_{n\times p}(\mathbb R),$$ $$\beta\in\mathcal M_{n\times 1}(\mathbb R ),$$ and $$\varepsilon\in\...
4
votes
2answers
768 views

Proving Convergence of Least Squares Regression with i.i.d. Gaussian Noise

I have a basic question that I can't seem to find an answer for -- perhaps I'm not wording it correctly. Suppose that we have an $n$-by-$d$ matrix, $X$ that represents input features, and we have a $n$...
4
votes
2answers
101 views

Correlation coefficient and regression line : Geometric intuition

correlation coefficient $$r = \frac{1}{n}\sum_{i=1}^n\frac{(x_i-\bar x)(y_i-\bar y)}{\sigma_x\cdot\sigma_y}$$ may be thought of as cosine of angle between two $n$-dimensional vectors $$ (x_1- \bar ...
4
votes
1answer
97 views

Why is $E[(Y-f(X))^2] = E[(Y-f(X))^T(Y-f(X))]$ in derivation of the regression function f?

I am working through Chapter 2 of The Elements of Statistical Learning (Hastie, Tibshirani, Friedman). I have problems following the authors as they analytically derive the regression coefficients $\...
4
votes
0answers
87 views

Convergence speed of discrete approximation

Here I asked the question about approximating the function $g(x) := \mathbb{E}(f(x,Y))$, where $x \in R$ and $Y$ is a random variable. If you follow the link you will see that $g(x)$ can be ...
4
votes
0answers
73 views

James–Stein estimator

Consider a FIR model of the form $y= Ug_0+e$ with $e$ white noise with variance $\sigma^2$. We assume that we have collected N input-output measurements $y$ and $U$. The James–Stein estimator is ...
4
votes
0answers
657 views

Leverage points in linear regression

From the wikipedia, leverage of a point is defined as the measure of how far away the independent variable values of an observation are from those of the other observations. Mathematically for point(...
3
votes
2answers
62 views

Are dependent variables random?

A linear regression model can be described as: $$ y = \beta_0 + \beta_1 X + \epsilon $$ where $\epsilon$ is the zero mean normal error. My question is: Is $X$ random variable? If no, then how can we ...
3
votes
1answer
477 views

Basic exponential regression

Background: I've been struggling with an exponential regression problem for about 8 months now (on and off): Vertically translated depreciation curve: Update the exponential regression coefficient ...
3
votes
1answer
83 views

(Linear Regression) Proving that Linear Regression is linear invariant

I want to ask how to show that Linear Regression is linear invariant? The problem is specified in the following picture: Here is the "solution" for the problem. But I really get confused by its ...
3
votes
2answers
1k views

why is the least square cost function for linear regression convex

I was looking at Andrew Ng's machine learning course and for linear regression he defined a hypothesis function to be $h(x) = \theta_0 + \theta_1x_1 + ... + \theta_nx_n$, where $x$ is a vector of ...
3
votes
2answers
2k views

Prove that $R^{2}$ cannot decrease when adding a variable

I know that in general this is true because the smaller model is nested within the larger model, so the larger model must have SSE at least as low as the smaller one, but I'm having a hard time ...
3
votes
1answer
84 views

Explain about the Correlation of Error Terms in Linear Regression Models

I would like to ask for the interpretation, both mathematically and intuitively if possible, about the homoscedasticity of the variance of errors in linear regression models. If there is correlation ...
3
votes
3answers
78 views

Basic math explanation (related to estimating linear regression with no intercept)

I have a question on the Cross Validated Stack Exchange site where I ask how to update the exponential regression coefficient of a vertically translated depreciation curve. A Cross Validated ...
3
votes
2answers
304 views

Regression Analysis (Line of Best Fit) for Categorical Variables

Brief Background/Motivation: I am looking at an Income vs. Education table that is adapted from a dissertation and was used in developing a curriculum in a social justice mathematics program. In the ...
3
votes
1answer
1k views

Why do we need gradient descent to minimize a cost function?

I read about regression in machine learning and came across this gradient descent algorithm to find the minimum value of a cost function. Then I read wikipedia to know more and it says the following ...
3
votes
1answer
212 views

Is there an unbiased estimator of the reciprocal of the slope in linear regression?

I have a situation which can be handled well through a simple linear regression model. That is, I have data points with known x values, y values with a given amount of error, and an ideal fit of the ...
3
votes
1answer
36 views

Question on lines of regression

I know how to find the line of regression when given the set of values of x and y. But in this question i don't have any idea what to do. I am a beginner . I will really appreciate the help.
3
votes
1answer
341 views

Linear Regression with Multiple Targets Derivation

I'm working through the derivation for the weights solution in multivariate linear regression, but keep getting tripped up when I try to solve for the case of multiple targets. Starting with the ...
3
votes
1answer
949 views

If $Y=X\beta+\epsilon$, prove that the least square estimator $\hat\beta$ is independent of $Y-X\hat{\beta}$

Let $Y=X\beta+\epsilon$, where $Y$ is an $n$ by $1$ vector, $X$ is an $n$ by $p$ matrix with full rank and $\epsilon$ is an $n$ by 1 vector of random errors independently and normally distribution ...
3
votes
1answer
274 views

What's the variance of intercept estimator in multiple linear regression?

Suppose a linear regression model $Y=Xβ+ε$ where $X$ is an $n$-by-$(k+1)$ matrix and $\epsilon$ follows $N(0,\sigma^2I_n)$. $k$ is the number of explanatory variables. The first column of $X$ is one (...
3
votes
2answers
105 views

Linear Regression Coefficients

Suppose you have a OLS linear regression with $\beta_0, \beta_1$, you data is {$x,y$}. Now, suppose we swap the labels of your data. $x\rightarrow y, y\rightarrow x.$ You get new $\beta':$ $\beta_0'...
3
votes
1answer
161 views

Variance of Beta in the Normal Linear Regression Model

Let $Y_1, Y_2, \ldots, Y_n$ represent response variables and let $x_1, x_2,\ldots, x_n$ be the associated explanatory variables. In the normal linear regression model, it's assumed that: $$Y_i \sim ...
3
votes
2answers
86 views

intuition behind having a unique regression line

I understand this mathematically. we have function of 2 variables represents the sum of square errors. We have to find the $a$ and $b$ that minimize the function. there is only one minimum point. But ...
3
votes
0answers
490 views

Bayesian Interpretation for Ridge Regression and the Lasso

I'm learning the book "Introduction to Statistical Learning" and in the Chapter 6 about "Linear Model Selection and Regularization", there is a small part about "Bayesian Interpretation for Ridge ...
3
votes
2answers
112 views

Ridge regression to minimize RMSE instead of MSE

Given a metrix $X$ and a vector $\vec{y}$, ordinary least squares (OLS) regression tries to find $\vec{c}$ such that $\left\| X \vec{c} - \vec{y} \right\|_2^2$ is minimal. (If we assume that $\left\| ...
3
votes
1answer
82 views

$\hat{Y} = X^T\hat{\beta}$ Matrix Dimension For Linear Regression Coefficients $\beta$

While reading about least squares implementation for machine learning I came across this passage in the following two photos: Perhaps I’m misinterpreting the meaning of $ \beta $ but if $ X^T$ has ...
2
votes
6answers
53 views

Solve for $\beta$. (Series)

I am proving the least squares estimates of the regression coefficients and I've come across these 2 equations. $$\sum_{i=1}^{n}y_i=\alpha n+\beta \sum_{i=1}^{n}x_i$$ $$\sum_{i=1}^{n}y_ix_i=\alpha \...
2
votes
2answers
2k views

Can a linear regression be quadratic?

The following is from a comp. sci. book that discusses regression. The passage seems to say that while a function fitted to a data set may be quadratic, it may yet be considered linear. This seems ...
2
votes
4answers
309 views

Does least squares (approximate solution) minimize the orthogonal distance of $b$ to $Ax$, or does it minimize the error projected along the $b$ axis?

I have always been confused about whether the approximate solution to $Ax=b$ is equivalent to minimizing the average distance of all of the $b$ vectors to $Ax$, or whether it is minimizing the ...
2
votes
2answers
75 views

Derivative transpose (follow up)

I'm working through the Elements of Statistical learning, and I have a quick followup to the below question: derivative transpose In the answer accepted, it states the below: $$ \frac{\partial}{\...
2
votes
2answers
548 views

Linear Regression: linear or reciprocal function?

The problem is given below: Simultaneous values of time $t$ and output $y$ from a specific sensor has been measured and is tabulated below $$\begin{array}{cc} t & y \\ \hline 1 & 17 \\ 2 ...
2
votes
1answer
121 views

Basic application of category theory to data science

Linear regression is the algorithm that, given a set of vectors ${\bf x}_i \in \mathbb{R}^p$, and a set of targets $y_i \in \mathbb{R}$, returns a vector ${\bf w} \in \mathbb{R}^p$ minimizing the ...
2
votes
2answers
55 views

Learn Noise / Error in Least Squares If We Know Its Form?

Let's say I am doing linear regression and I have a data matrix $A$. And, I know the noise $e_i$, is zero mean (and perhaps we know the distribution, too): $$y_i = a_i^T x + e_i$$ Obviously from ...
2
votes
1answer
137 views

Variance Estimate in linear regression

In a linear regression, $y=X\beta+\epsilon$, where $\epsilon\sim N(0, \sigma^2)$, $X\sim R^{N \times (p+1)}$. Assume the observations $y_i$ are uncorrelated and have constant variance $\sigma^2$, and ...
2
votes
1answer
2k views

derivative transpose

I'm reading the book "The Elements of Statistical Learning - Data Mining, Inference, and Prediction" chapter 3 and there comes a simple derivation that I don't understand: We have: $...
2
votes
2answers
33 views

Relationship between OLS estimates of slope coefficients of simple linear regression Y on X and X on Y

Assume a model $y = \beta_0 + \beta_1x + u$. Given a sample $(x_i, y_i)_{i=1}^n$, we can find the OLS estimates of $\beta_1$, $\hat{\beta_1}$. Then suppose that we assume another model $x = \gamma_0 + ...
2
votes
1answer
134 views

how fit a model with data following asymptotic / sigmoid pattern

I'm trying to fit data. I assume that the association between dependent and indepdent variable is of the form $$T(y)=aR(x)+b$$ I also know that my data are ressemble either an asymptotic function ...
2
votes
2answers
41 views

Why is a term that comes out of a variance bracket is squared?

I am in a course on data analysis. The following statement is made in the notes made available to us by our professor: $$ \text{Var}[a] = \text{Var}[\bar{y} -b\bar{x}] = \text{Var}[\bar{y}] + \text{...
2
votes
1answer
2k views

The cost function derivation in andrew ng machine learning course

Can you please give me how did Andrew NG, came up with the formula for cost function $$J(\theta_0,\theta_1) = \frac{1}{2m}\sum_{i=0}^m{(H_\theta(x^i)-y^i))^2}$$ AFAIK the square is being taken to ...
2
votes
1answer
2k views

Objective function of linear regression problem with regularization

We have the following: The design matrix $X \in R^{n \times d}$ The output vector $y \in R^n$ The weight vector $w \in R^d$ Let $T = \tau I_d$, where $I_d$ is the $d \times d$ identity matrix and $\...