1
$\begingroup$

I have been taught to calculate the estimates $\beta_0$ and $\beta_1$ (or $\vec{\beta}$) by using the following formula

$$\sum_{i=1}^{n} y_i \vec{X_i}^T = \vec{\beta}^T\sum_{i=1}^{n} X_i \vec{X_i}^T$$

First of all, it is important to understand that a linear regression model of the form

$$y_i = \beta_0 + \beta_1 x_i + \epsilon_i$$ for $i=1...n$ (i.e. we have $n$ linear equations) can be represented in a matrix notation as follows $$\vec{y} = X \vec{\beta} + \vec{\epsilon}$$ where, in this case, $$X = \begin{pmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n\end{pmatrix}$$ and $$\vec{\beta} = \begin{pmatrix} \beta_0 \\ \beta_1 \end{pmatrix}$$

Now, I have an exercise where it is given $X_i = [1, x_i]$ (and $x_i$ is given). Nothing special so far, since $X_i$ should represent the $ith$ row vector of the matrix $X$. But when I try to calculate the covariance matrix $$\sum_{i=1}^{n} X_i \vec{X_i}^T$$ as follows $$[1, x_1] \begin{pmatrix} 1 \\ x_1 \end{pmatrix} + ... + [1, x_n] \begin{pmatrix} 1 \\ x_n \end{pmatrix}$$

Well, since $[1, x_1]$ is a $1$ by $2$ vector and $\begin{pmatrix} 1 \\ x_1 \end{pmatrix}$ a $2$ by $1$ vector, the resulting product is a scalar, but what I am computing is a matrix, i.e. the covariance matrix. In other words, to obtain a matrix, we need to switch the multiplicands.

What am I missing? I think it's the notation that's not clear. For example I would interpret $[a, b]$ as a column vector instead of a row vector like in my case above. But in my case above I need to interpret $X_i = [1, x_1]$ as a row vector because $X_i$ is actually a row from $X$.

$\endgroup$
2
  • $\begingroup$ Are you looking to calculate the covariance matrix of the vector $\boldsymbol{\beta}$? $\endgroup$ Commented Jan 17, 2016 at 18:45
  • $\begingroup$ It should be a row vector. $\endgroup$ Commented Jan 17, 2016 at 18:51

1 Answer 1

2
$\begingroup$

If you use the formula $$y_i = \beta_0 + \beta_1 x_i + \epsilon_i$$ and $$\vec{y} = X \vec{\beta} + \vec{\epsilon}$$ where $$X = \begin{pmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n\end{pmatrix}$$

Then the first formula should be $$\vec{X}^T\vec{y}=\vec{X}^T\vec{X}\vec{\beta}$$

So

$$\vec{X}^T\vec{X}=\begin{pmatrix} 1&1&\dots&1\\ x_1&x_2&\dots&x_n\end{pmatrix}\begin{pmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n\end{pmatrix}$$

Edit:

I just realized that we can get the same result using their notation, with $X_i$ representing the column vector $\begin{pmatrix}1\\x_i\end{pmatrix}$. Continuing your formula: $$\begin{pmatrix} 1 \\ x_1 \end{pmatrix}[1, x_1] + ... + \begin{pmatrix} 1 \\ x_n \end{pmatrix}[1, x_n] =\begin{pmatrix} 1 &x_1\\ x_1&x_1^2 \end{pmatrix}+\dots+\begin{pmatrix} 1 &x_n\\ x_n&x_n^2 \end{pmatrix}=\begin{pmatrix} n &\sum x_i\\ \sum x_i&\sum x_i^2 \end{pmatrix}$$

This is exactly what you get from the big matrices multiplication.

$\endgroup$
0

You must log in to answer this question.