4
$\begingroup$

In linear regression, $$y = X \beta + \epsilon$$ where $y$ is a $n \times 1$ vector of observations for the response variable,

$X = (x_{1}^{T}, ..., x_{n}^{T}), x_{i} \in \mathbb{R}^p. i =1,...,n$ is a data matrix of $p$ explanatory variables, and $\epsilon$ is a vector of errors.

Further, assume that $\mathbb{E}[\epsilon_i] = 0$ and $var(\epsilon_i) = \sigma^2, i=1,...n$

The least-squares estimate, $$\hat{\beta} = (X^{T}X)^{-1}X^{T}y$$

The least-squares estimators are the fitted values, $$\hat{y} = X \hat{\beta} = X(X^{T}X)^{-1}X^{T}y = X C^{-1}X^{T}y = Py$$

$P$ is a projection matrix. It is has the following properties:

  1. idempotent, meaning P*P = P
  2. symmetric
  3. positive semi-definite

For property 1, what's the intuition behind this? How can you take some matrix do transformation, inverse and multiplication, then, you get idempotent. It's an important concept. But, it's hard to follow through the math to get an intuition.

Why will we get property 2 and property 3, How am I supposed to think about this?

$\endgroup$
3
  • $\begingroup$ I'm not sure I follow completely what your question is. Property 1 can be verified by simply calculating $P^2$. Property 2 and 3 in a similar fashion. $\endgroup$ – Gregory Dec 23 '17 at 20:56
  • $\begingroup$ If the question is why does this happen, I'm not exactly sure. $\endgroup$ – Gregory Dec 23 '17 at 20:57
  • 2
    $\begingroup$ Idempotent goes hand-in-hand with projection en.wikipedia.org/wiki/Projection_(linear_algebra) $\endgroup$ – user365239 Dec 23 '17 at 20:58
3
$\begingroup$

I believe you’re asking for the intuition behind those three properties of the hat matrix, so I’ll try to rely on intuition alone and use as little math and higher level linear algebra concepts as possible.

Preliminaries

Start with the fact that the projection matrix $P$ allows you to obtain the orthogonal projection of an arbitrary vector onto the column space of X. Let’s use $v_p$ for the orthogonal projection of $v$: $$ P v = v_p $$ You can use $P$ to decompose any vector $v$ into two components that are orthogonal to each other. Think of $v_n$ as what is "left over" after the rest of $v$ is projected onto the column space of X, so it is orthogonal to the column space of X (and any vector in the column space of X). $$ v = v_p + v_n $$ $$ v_p \perp v_n $$

1. Why does P * P = P?

Intuitively, projecting a vector onto a subspace twice in a row has the same effect as projecting it onto that subspace once. The second projection has no effect because the vector is already in the subspace from the first projection.

Less intuitive

If that isn’t intuitive, it may be easier to consider the equivalent question: why does $P * P v= P v$ for any arbitrary vector v?

Start by simplifying the left hand side: $$ P * (P v) = P v_p $$ since $P v = v_p$.

Next consider $ P v_p $, which (by definition of P) projects $v_p$ onto the column space of X. This has no effect since $v_p$ is already entirely in the column space of X. Therefore
$$ P v_p = v_p $$ Since $v_p = P v$, we conclude: $$ P v_p = P v $$ Chaining all these equations together gives: $$ P * P v= P v $$

2. Why is P symmetric?

Intuitively, consider two arbitrary vectors $v$ and $w$. Take the dot product of one vector with the projection of the other vector. $$ (P v) \cdot w $$ $$ v \cdot (P w) $$

In both dot products, one term ($P v$ or $P w$) lies entirely in the ‘projected space’ (column space of X), so both dot products ignore everything that is not in the column space of X. This means both dot products are equal. Some simple dot product identities then imply that $P = P^T$, so $P$ is symmetric.

Less intuitive

If that isn't intuitive, we first prove that both dot products are equal. Decompose $v$ and $w$ as shown in the preliminaries above. $$ v = v_p + v_n $$ $$ w = w_p + w_n $$ The projection of a vector lies in a subspace. The dot product of anything in this subspace with anything orthogonal to this subspace is zero. We use this fact on the dot product of one vector with the projection of the other vector: $$ (P v) \cdot w \hspace{1cm} v \cdot (P w) $$ $$ v_p \cdot w \hspace{1cm} v \cdot w_p $$ $$ v_p \cdot (w_p + w_n) \hspace{1cm} (v_p + v_n) \cdot w_p $$ $$ v_p \cdot w_p + v_p \cdot w_n \hspace{1cm} v_p \cdot w_p + v_n \cdot w_p $$ $$ v_p \cdot w_p \hspace{1cm} v_p \cdot w_p $$ Therefore $$ (Pv) \cdot w = v \cdot (Pw) $$ Next, we can show that a consequence of this equality is that the projection matrix P must be symmetric. Here we begin by expressing the dot product in terms of transposes and matrix multiplication (using the identity $x \cdot y = x^T y$ ): $$ (P v) \cdot w = v \cdot (P w) $$ $$ (P v)^T w = v^T (P w) $$ $$ v^T P^T w = v^T P w $$ Since v and w can be any vectors, the above equality implies: $$ P^T = P $$

3. Why is P positive semidefinite?

By definition a matrix $P$ is positive semidefinite if and only if for every non-zero column vector $v$: $$ v^T P v >= 0 $$ or equivalently: $$ v \cdot (P v) >= 0 $$ Intuitively, a dot product is a projection of one vector onto another vector, and then scaling by the length of the second vector. We want to show that this dot product is non-negative. In the equation immediately above, $v \cdot (P v)$ means "project $v$ onto $P v$ and scale by $P v$". The first part of this, project $v$ onto $P v$, is equivalent to "project $v$ onto $v_p$", since $P v = v_p $.

Projecting $v$ onto $v_p$ projects $v$ onto something that lies entirely in the column space of X, so this projection is just $v_p$. Next, scaling this $v_p$ by $v_p$ squares its length. A squared length must be non-negative.

Less intuitive

If that isn't intuitive, the dot product can be simplified by decomposing $v$ into orthogonal components $$ v \cdot (P v) $$ $$ (v_p + v_n) \cdot (P v) $$ $$ (v_p + v_n) \cdot v_p $$ $$ v_p \cdot v_p + v_n \cdot v_p $$ Since $v_p$ and $v_n$ are orthogonal, the second term is zero and we have only $$ v_p \cdot v_p $$ The quantity immediately above is the length of the vector $v_p$ squared (i.e., $\|v_p\|_2^2$ ). This must be a non-negative value. $$ v_p \cdot v_p = \|v_p\|_2^2 >= 0 $$

$\endgroup$
0
$\begingroup$

Basically when you have $n$ observations with $p$ unknown coefficients where $n> p$, it means that you have an over-determined system of equations, that is $$ X\beta = y. $$ Namely, every subset of $p$ equations will give you another set of $\hat{\beta}$. Hence, you cannot just solve this system of equation rather you have to find an approximate solution. Such a solution lives in the column space of $X$ (as every solution of $Ax=b$ belongs, by definition, to the column space of $A$). Now, you are interested in the "best" solution, namely, a vector $\beta$ that will solve the modified system of linear equations. Namely, instead of solving $X\beta = y$ you can solve $X'X\beta = X'y$ which will have a unique solution w.r.t. $\beta$ and instead of the original $y$ you will have $\hat{y}$ that is a vector that belongs to $C(X)$ and it is the "closest" possible vector in $C(X)$ to the original $y$.

Why it is the best? A matrix $P=A(A'A)^{-1}A'$ is a projection matrix into the column space of $A$ (why it has this specific form you can read in the link that is given in the comments). Why does $P^2 = P$? Let us see what does $X(X'X)^{-1}X'$ to$x$, where $x \in C(X)$. If $x$ is already in the column space of $X$ thus "projecting" it on $C(X)$ will do nothing, i.e., will return $x$ itself. $P^2 = PP$ is in a sense like projecting set of vectors from $C(X)$ onto $C(X)$, hence you should get $P$ itself.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.