0
$\begingroup$

On the image there is the empirical regression line (the best fit by sum of min squares - $y_i - \hat{y}$).

But how would the theoretical regression line look like? Does it join all the points? I guess, then it would not be line.

Thanks

$\endgroup$

2 Answers 2

0
$\begingroup$

Assuming the points are generated by adding i.i.d. Gaussian noise $\epsilon_i$ to the true regression line, you cannot know what the true regression line is without more information, e.g., knowing what the actual value of the Gaussian noise terms $\epsilon_i$. After all, that is the point of regression; if we knew what the true regression line were, we wouldn't need to do least squares.

$\endgroup$
0
$\begingroup$

Note that the real regression line is the conditional expectation, i.e., Let's say that you are observing $\{(X_i, Y_i)\}_{i=1}^n$, and you are interested to predict $Y$ using $X$, as such by the orthogonal decomposition $$ Y= E[Y|X]+\epsilon. $$ Note that $E[\epsilon|X]=E[Y - E[Y|X] |X]=0$, thus by using regression you are trying to estimate the conditional mean $E[Y|X]$. If $(Y,X)$ is $MVN(\mu, \Sigma)$, then the error term is Guassian and the conditional mean is indeed a straight line $$ E[Y|X] = \beta_0 + \beta_1 X, $$ where $\beta_0 = \mu_X - \beta_1 \mu_X$ and $\beta_1 = cov(X,Y)/\sigma^2_X$. For non Gaussian setting, the conditional mean can be non-linear, thus the regression $$ Y = \hat{\beta}_0 + \hat{\beta}_1X, $$ is only a linear approximation of $E[Y|X]=g(X)$ which quality depends on the structure of $g$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .