In the linear regression model $Y = X\beta + e$, where $Y, e$ are $n \times 1$ random vectors, $X$ is an $n \times k$ random matrix and $\beta$ a $k \times 1$ vector of parameters, it is sometimes said that under the assumption $e \mid X \sim N(0, \sigma^2 I_n)$ one has $\hat{\beta} - \beta \mid X \sim N\big(0, \sigma^2(X^TX)^{-1}\big)$, where $\hat{\beta} := (X^TX)^{-1}X^TY$ is the OLS estimate for $\beta$.

Usually this is justified by applying the proposition "if $Z\sim N(0, \Sigma)$ and $W:= AZ$ with $A\Sigma A^T > 0$ then $W \sim N(0, A\Sigma A^T)$", since a standard result in OLS regression (under suitable assumption) is:

$\hat{\beta} - \beta = (X^TX)^{-1}X^{T}e$

I am not quite satisfied with the above justification. When I see $e\mid X \sim N(0, \sigma^2I_n)$ I interpret it just as a short hand notation for:

$f_{e \mid X}(u \mid x) = \frac{1}{\sqrt[n]{2 \pi} \sigma^{n}}\exp\Big(-\frac{u^Tu}{2\sigma^2}\Big)$

I guess my problem is that I am not used to treating $e\mid X$ as a random variable.

What I had in mind is something along the lines: begin with the conditional distribution $F_{\hat{\beta} - \beta \mid X}(u \mid x)$, arrive (somehow) to a known distribution and take derivatives. When I try to do it I get:

$F_{\hat{\beta} - \beta \mid X}(u \mid x) = \Pr(\hat{\beta} - \beta \leq u \mid X = x) = \Pr\big((X^TX)^{-1}X^Te \leq u \mid X = x\big)$

but I don't see how to proceed from there. So, my questions are:

  1. In what sense is $e \mid X$ a random variable? [Yes, $f_{e\mid X}(u, x)$ is a density function but I am used to thinking random variables as functions from one measurable space to another].

  2. Is there a way to make the above attempt at a proof, work?


  1. Take a r.v $Y$ with $E|Y|^2 < \infty$ and another random variable $X$ , ($E|X|^2 < \infty$) on the same probability space, then $Y$ can be orthogonaly decomposed into $$ Y = E[Y|X]+ e, $$ from the decomposition it follows that $$ E[e|X]=0, $$ however any parametric specification is mainly based on ones' assumptions. In particular, one can assume that $e|X\sim N(0, \sigma^2)$. This is the stochastic model that generates your observed data points $\{(y_i, x_i)\}$. So, basically you have an inverse problem when your goal is reconstruct $E[Y|X]$. If $(Y,X)$ is bivariate normal then the OLS estimators provide consistent estimators, otherwise it is just a linear approximation.

  2. Every entry of your vector is a an integral over normal density function. Generally, there is no close form for it, but you can still take derivatives and get the multivariate normal distribution,


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.