# how to find the distribution of the least square estimator $\beta$?

i'm solving a problem that involve a linear model, and i'm trying to get the distribution of the least square estimator $$\beta$$.

i found in a book that:

$$\widehat{\beta}\sim N_{p}(\beta, (X^{\prime}X)^{-}\sigma^{2})$$. but how can i get the distributions to $$\beta_{i}$$?.

in my case, the matrix $$(X^{\prime}X)^{-}$$, i found was:

$$\frac 1 \alpha \begin{bmatrix} \sum_{i=1}^N x_i^2 & -\sum_{i=1}^N x_i \\ & \\ -\sum_{i=1}^N x_i & N\\ \end{bmatrix}$$

where $$\alpha= N \sum_{i=1}^N x_i^2- \left[ \sum_{i=1}^N x_i \right]^2$$ and i'm looking for the distribution of $$\beta_1, \beta_2$$.

• For your model you have $N$ observations, the intercept is $\beta_1$, and the slope is $\beta_2$, correct? Also your inverse matrix $(X'X)^{-1}$ should have a factor $$\frac1{N\sum x_i^2-(\sum x_i)^2}$$ in front of it. – grand_chat Sep 22 '18 at 1:19
• yes, sorry you are right. i have edited. But i'm still confuse cuz $\beta$ is a vector and $(X^{\prime}X)^{-} \sigma^{2}$ is a matrix. so what is the variance of $\beta$'s. – JohanR Sep 22 '18 at 1:43

The result that you found $$\hat\beta\sim N_p(\beta, (X'X)^{-1}\sigma^2)\tag{*}$$ allows you to read off the solution to your problem. If your model is $$Y_i = \beta_1 + \beta_2x_i+\varepsilon_i,\qquad i=1,\ldots,N,$$ then interpret the assertion (*) to mean that the joint distribution of $$(\hat\beta_1,\hat\beta_2)$$ is bivariate normal ($$p=2$$) with mean $$(\beta_1,\beta_2)$$ and covariance matrix $$(X'X)^{-1}\sigma^2$$. So the marginal distribution of $$\hat\beta_1$$ is normal with mean $$\beta_1$$ and variance equal to the $$(1,1)$$ entry of the covariance matrix, which is $$\frac{\sigma^2}{N\sum x_i^2-(\sum x_i)^2}\sum x_i^2=\sigma^2\frac{\sum x_i^2}{N\sum x_i^2-(\sum x_i)^2}.\tag1$$ Similarly the distribution of $$\hat\beta_2$$ is normal with mean $$\beta_2$$ and variance equal to the $$(2,2)$$ entry of the covariance matrix, which is $$\frac{\sigma^2}{N\sum x_i^2-(\sum x_i)^2}N=\sigma^2\frac{N}{N\sum x_i^2-(\sum x_i)^2}.\tag2$$ In (1) and (2), the denominator can be rewritten $$N\sum(x_i-\bar x)^2$$.
If your model is $$Y_i = \theta x_i + \varepsilon_i,\qquad i=1,\ldots,N,$$ then the design matrix $$X$$ is a column $$[x_1, x_2,\ldots,x_N]'$$ so $$X'X=\sum x_i^2$$ and the result (*) says that the estimator $$\hat\theta$$ is normal with mean $$\theta$$ and variance $$\displaystyle\frac{\sigma^2}{\sum x_i^2}.$$
• @JohanR Please see my edit. It is fine to define the $x_i=t_i^2/2$ as you have done, assuming the $t_i$'s are nonrandom. Your model is now $Y_i=\theta t_i+\varepsilon_i$ with no intercept. It has only one parameter. – grand_chat Sep 22 '18 at 3:11
$$\widehat \beta_1 = \big[ 1, 0\big] \widehat\beta.$$ Therefore \begin{align} & \operatorname E\left(\widehat\beta_1\right) = \big[1,0\big]\operatorname E \left( \widehat\beta \right) = \big[1,0\big]\beta = \beta_1. \\[10pt] & \operatorname{var}\left(\widehat\beta_1\right) = \operatorname{var}\left( \big[1,0\big] \widehat\beta \right) = \big[1,0\big]\Big( \operatorname{var}\left(\widehat\beta\right) \Big) \left[ \begin{array}{c} 1 \\ 0 \end{array} \right] \\[10pt] = {} & \sigma^2 \big[1,0\big] \left( \frac 1 \alpha \begin{bmatrix} \sum_{i=1}^N x_i^2 & -\sum_{i=1}^N x_i \\ & \\ -\sum_{i=1}^N x_i & N\\ \end{bmatrix} \right) \left[ \begin{array}{c} 1 \\ 0 \end{array} \right] = \cdots \end{align} That gives you the expected value and the variance. And if $$\widehat\beta$$ is bivariate normal then a linear combination of its components with constant (i.e. non-random) coefficients is univariate normal.
And similarly for $$\widehat\beta_2.$$