Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm studying time series and in one of my resources I encountered the following:

The linear model described by $\text{(2.1)}$ above can be conveniently written in a more general notation by defining the column vectors $\boldsymbol{z}_t=(x_{t1},x_{t2},\ldots,x_{tq})'$ and $\boldsymbol{\beta}=(\beta_1,\beta_2,\ldots,\beta_q)'$, where $'$ denotes transpose, so $\text{(2.1)}$ can be written in the alternate form $$x_t=\boldsymbol{\beta'}\boldsymbol{z}_t+w_t.\tag{2.2}$$ where $w_t\sim \text{iid}\,\,\text{N}(0,\sigma_w^2)$. It is natural to consider estimating the unknown coefficient vector $\boldsymbol{\beta}$ by minimizing the error sum of squares $$Q=\sum_{t=1}^nw_t^2=\sum_{t=1}^n(x_t-\boldsymbol{\beta'}\boldsymbol{z}_t)^2,\tag{2.3}$$ with respect to $\beta_1,\beta_2,\ldots,\beta_q$. Minimizing $Q$ yields the ordinary least squares estimator of $\boldsymbol{\beta}$ . This minimization can be accomplished by differentiating $\text{(2.3)}$ with respect to the vector $\boldsymbol{\beta}$ or by using the properties of projections. In the notation above, this procedure gives the normal equations $$\left(\sum_{t=1}^n\boldsymbol{z}_t\boldsymbol{z}'_t\right)\widehat{\boldsymbol{\beta}}=\sum_{t=1}^n\boldsymbol{z}_tx_t.\tag{2.4}$$ The notation can be simplified by defining $Z\,=\,[\boldsymbol{z}_1|\boldsymbol{z}_2|\cdots|\boldsymbol{z}_n]'$ as the $n\times q$ matrix composed of the $n$ samples of the input variables, the observed $n\times 1$ vector $\boldsymbol{x}\,=\,(x_1,x_2,\ldots,x_n)'$ and the $n\times 1$ vector of errors.

The normal equations, $\text{(2.4)}$, can be written as $$(Z'Z)\widehat{\boldsymbol{\beta}}=Z'\boldsymbol{x}\tag{2.6}$$ and the solution $$\widehat{\boldsymbol{\beta}}=(Z'Z)^{-1}Z'\boldsymbol{x}\tag{2.7}$$ when the matrix $Z'Z$ is nonsingular. The minimized error sum of squares $\text{(2.3)}$, denoted $SSE$, can be written as $$\begin{align} SSE&=\sum_{t=1}^n(x_t-\widehat{\boldsymbol{\beta}}'\boldsymbol{z}_t)^2\\ &=(\boldsymbol{x}-Z\widehat{\boldsymbol{\beta}})'(\boldsymbol{x}-Z\widehat{\boldsymbol{\beta}})\\ &=\boldsymbol{x}'\boldsymbol{x}-\widehat{\boldsymbol{\beta}}'Z'\boldsymbol{x}\\ &=\boldsymbol{x}'\boldsymbol{x}-\boldsymbol{x}'Z(Z'Z)^{-1}Z'\boldsymbol{x},\\ \end{align}\tag{2.8}$$ to give some useful versions for later reference. The ordinary least squares estimators are unbiased, i.e.e, $E(\widehat{\boldsymbol{\beta}})=\boldsymbol{\beta}$, and have the smallest variance within the class of linear unbiased estimators.

If the errors $w_t$ are normally distributed, $\widehat{\boldsymbol{\beta}}$ is also the maximum likelihood estimator for $\boldsymbol{\beta}$ and is normally distributed with $$\color{red}{\boxed{\displaystyle\color{black}{\text{cov}(\widehat{\boldsymbol{\beta}})=\sigma_w^2\left(\sum_{t=1}^n\boldsymbol{z}_t\boldsymbol{z}'_t\right)^{-1}=\sigma_w^2(Z'Z)^{-1}=\sigma_w^2C,}}}\tag{2.9}$$ where $$C=(Z'Z)^{-1}\tag{2.10}$$ is a convenient notation for later equations.

The part which I have marked is the one I don't understand?! that autocovariance or what? Can someone write the formula more explicitly or something?

Thnx =)

share|cite|improve this question
Do you know matrix algebra? Do you know the linear regression model? This is a result about simple regression and the quantity as written here either could or could not be related to autocovariances depending on what is inside $z_t$. – Learner Jan 22 '13 at 12:08
Yes I do know =) But for some reason I don't understand that notation... – jjepsuomi Jan 22 '13 at 12:14
Please take a look at the formula for the variance matrix here… (second formula). This is a general result. Maybe if you try to think about it in that simpler context, things could get clearer. – Learner Jan 22 '13 at 12:25
up vote 1 down vote accepted

In the red box you have (the inverse of) a matrix. The reason is that the vector (I suppress the $t$) $z:=(z_1,\dots,z_n)$ and $z^t:=(z_1,\dots,z_n)^t$ are multiplied w.r.t. the Kronecker product. You can find its definition at Kronecker Product.

The idea is to look at $z$ as a $1\times n$ matrix, while $z^t$ is a $n\times 1$ matrix. By definition, their product is a

$$(1\cdot n)\times (n\cdot 1)= n\times n$$


share|cite|improve this answer
+1 And thank you again @Avitus :) I truly appreciate for your effort answering...I think it is very important thing and what I will keep in mind when helping others myself :) – jjepsuomi Aug 5 '13 at 9:40
@jjepsuomi thanks for the kind words: you are welcome and...keep going :-) – Avitus Aug 12 '13 at 8:26

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.