4
$\begingroup$

My teacher has given me the following definition of the canonical spectrum theorem:

Let a matrix the $A \in M_{n\times n}(\mathbb{R})$, and set of eigenvalues, $\sigma(A)$={$\lambda_1$,$\lambda_2$........,$\lambda_k$}.

$A$ is diagonalisable $\Leftrightarrow A = \sum{\lambda_ip_i}$ such that

  1. $p_i:$ orthogonal projection onto $\mathcal{E}_{\lambda_i(A)},$ the eigen space

  2. $p_ip_j = 0 \space \text{if} \space i\neq j$

  3. $\sum{p_i}=I,$ the identity matrix.

My question is: I don't understand above 3 points, what does mean of orthogonal projection here, and how it is related with eigen space? Also point 2 and 3 don't understand. Anybody help me to understand by giving example of above 3 points.

$\endgroup$
10
  • 1
    $\begingroup$ The orthogonal projection is a transformation mapping a vector to the unique closest vector in the subspace with respect to the norm induced by the inner product (usually the $L^2$ norm). $\endgroup$ Apr 19 at 0:13
  • 2
    $\begingroup$ Item 2 is wrong. They are orthogonal projections, so the matrices multiply to give $0$. $\endgroup$ Apr 19 at 1:58
  • 1
    $\begingroup$ @CameronWilliams see now,it's corrected, $\endgroup$
    – David
    Apr 19 at 2:01
  • 1
    $\begingroup$ Either you are misquoting your teacher or they were wrong. Every orthogonal projection matrix is real symmetric. If $A=\sum_i\lambda_ip_i$, $A$ must be symmetric, but this is not given in the theorem. $\endgroup$
    – user1551
    Apr 19 at 2:14
  • 2
    $\begingroup$ @JonathanZ Oh, indeed, it is only true for eigenspaces. I should have been more careful. $\endgroup$ Apr 19 at 3:08

2 Answers 2

6
$\begingroup$

The spectral theorem says that for a (symmetric/hermitian/normal) matrix, we can find projection $(PP=P)$ matrices $P_1,\dots,P_n$ such that $A$ has the representation

$$A = \lambda_1 P_1 + \dots + \lambda_n P_n$$

where

  1. Each of the $P_i$ are projection matrices. The orthogonal projection is a transformation mapping a vector to the unique closest vector in the $\lambda_i$ eigenspace with respect to the norm induced by the inner product (usually the $L^2$ norm).
  2. The $P_iP_j=0$ $i\neq j$
  3. The $P_i$ are a "resolution of the identity" meaning they sum to the identity matrix.

Aside: One cool thing that we can do from here is define a function of a matrix as

$$f(A):= f(\lambda_1)P_1 + \dots + f(\lambda_n)P_n$$


Example of the theorem holding.

Let's take a nice symmetric matrix so we know the spectral theorem holds.

$$ \left(\begin{array}{ll} 2 & 1 \\ 1 & 2 \end{array}\right) $$

The eigenvalues are

$$\lambda_1 = 3, \lambda_2 = 1$$

and the (normalized) eigenvectors are

$$ v_1=\frac{1}{\sqrt{2}}\left[\begin{array}{l} 1 \\ 1 \end{array}\right], \quad v_2=\frac{1}{\sqrt{2}}\left[\begin{array}{c} -1 \\ 1 \end{array}\right]. $$

Now we compute the projections onto the eigenspaces.

$$P_1 = v_1v_1^T = \left[\begin{array}{ll} \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & \frac{1}{2} \end{array}\right] $$

and $$P_2 = v_2v_2^T = \left[\begin{array}{ll} \frac{1}{2} & \frac{-1}{2} \\ \frac{-1}{2} & \frac{1}{2} \end{array}\right] $$

  1. Verifying the first claim, we see that $$3 \left[\begin{array}{ll} \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & \frac{1}{2} \end{array}\right] + 1\left[\begin{array}{ll} \frac{1}{2} & \frac{-1}{2} \\ \frac{-1}{2} & \frac{1}{2} \end{array}\right] = A$$
  2. Checking the second, we have $$\left[\begin{array}{ll} \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & \frac{1}{2} \end{array}\right]\left[\begin{array}{ll} \frac{1}{2} & \frac{-1}{2} \\ \frac{-1}{2} & \frac{1}{2} \end{array}\right] = \left[\begin{array}{ll} 0 & 0 \\ 0 & 0 \end{array}\right]$$
  3. And for the third, $$\left[\begin{array}{ll} \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & \frac{1}{2} \end{array}\right]+\left[\begin{array}{ll} \frac{1}{2} & \frac{-1}{2} \\ \frac{-1}{2} & \frac{1}{2} \end{array}\right] = \left[\begin{array}{ll} 1 & 0 \\ 0 & 1 \end{array}\right]$$

Which shows with a specific example, the three assertions.


Counter-example of the theorem NOT holidng: Here is a matrix for which the spectral theorem does not hold because the matrix is not normal (i.e. $AA^*\neq A^*A)$

$$ A=\left(\begin{array}{cc} 6 & -7 \\ 0 & 3 \end{array}\right) $$

the matrix is not normal because

$$ \left(\begin{array}{cc} 6 & 0 \\ -7 & 3 \end{array}\right) \cdot\left(\begin{array}{cc} 6 & -7 \\ 0 & 3 \end{array}\right)=\left(\begin{array}{cc} 36 & -42 \\ -42 & 58 \end{array}\right) \neq\left(\begin{array}{cc} 85 & -21 \\ -21 & 9 \end{array}\right)=\left(\begin{array}{cc} 6 & -7 \\ 0 & 3 \end{array}\right) \cdot\left(\begin{array}{cc} 6 & 0 \\ -7 & 3 \end{array}\right) $$

Its eigenvalues are $$\lambda_1=6, \lambda_2 = 3$$

and its corresponding (normalized) eigenvectors are $$ v_1= \left[\begin{array}{l} 1 \\ 0 \end{array}\right] , v_2=\frac{1}{\sqrt{7^2+3^2}}\left[\begin{array}{l} 7 \\ 3 \end{array}\right] $$

We compute the projections onto the eigenspaces

$$P_1 = v_1v_1^T = \begin{bmatrix} 1& 0 \\ 0 & 0 \end{bmatrix} $$

$$P_2 = v_2v_2^T = \frac{1}{58}\begin{bmatrix} 49& 21 \\ 21 & 0 \end{bmatrix} $$

but we see that $A\neq 6P_1 + 3P_2$ and this is because the spectral theorem does not hold for a non-normal matrix. Moreover, we see that $P_1 + P_2 \neq I$ which is again a consequence of the matrix not being normal.


Requested insights about projections:

A projection matrix/operator is one such that $P^2=P$.

In words: projecting a second time does nothing because its already projected.

The eigenvalues of a projection matrix are contained in $\{0,1\}$. Suppose $v$ is an eigenvector. Then $$\lambda^2 v = \lambda Pv = PPv = Pv = \lambda v$$

so $\lambda^2=\lambda$ but this implies that $\lambda\in \{0,1\}$. This also easily follows from the spectral mapping theorem.

Ok so where do these types of operators appear? Well consider the following problem:

For a subspace $W\subset \mathbb{R}^n$, and a vector $v\in \mathbb{R}^n$, what is

$$\operatorname{argmin}_{w\in W}\|v-w\|$$

i.e. what are all the vectors in $W$ who have minimal distance to $v$. In an inner product space, it turns out that the answer to this question, is that there is one unique vector!

Now we define the following operator

$$P_Wv := \operatorname{argmin}_{w\in W}\|v-w\|$$

which is well defined because the right hand side exists and is unique. This operator will be a projection operator $(P_W^2=P_W)$ and $P_W:\mathbb{R}^n \to W$.

Now you might be wondering, how do we compute these projection matrices? Well it's actaully pretty easy. In linear algebra class, there are traditionally two problem types students do to compute these.

  • Projecting a vector $v$ onto the span of another vector $W=\operatorname{span}{w}$
  • Projecting a vector $v$ onto the span of two vectors in $\mathbb{R}^3$ $W=\operatorname{span}{w_1,w_2}$

I would suggest refreshing yourself on how to complete both of these types of problems from elementary linear algebra if you don't remember.

But essentially, in finite dimensions, all of this can be boiled down to the following table from Gilbert Strang (5th edition (not 4th) page 206.

  • The projection of a vector $b$ onto the line through $a$ is the closest point $p=a\left(a^{\mathrm{T}} b / a^{\mathrm{T}} a\right)$
  • The error $\boldsymbol{e}=\boldsymbol{b}-\boldsymbol{p}$ is perpendicular to $a$ : Right triangle $\boldsymbol{b} \boldsymbol{p} \boldsymbol{e}$ has $\|p\|^2+\|\boldsymbol{e}\|^2=\|b\|^2$.
  • The projection of $b$ onto a subspace $S$ is the closest vector $p$ in $S ; b-p$ is orthogonal to $S$.
  • $A^{\mathrm{T}} A$ is invertible (and symmetric) only if $A$ has independent columns: $N\left(A^{\mathrm{T}} A\right)=N(A)$.
  • Then the projection of $b$ onto the column space of $A$ is the vector $p=A\left(A^{\mathrm{T}} A\right)^{-1} A^{\mathrm{T}} b$.
  • The projection matrix onto $C(A)$ is $P=A\left(A^{T} A\right)^{-1} A^{\mathrm{T}}$. It has $\boldsymbol{p}=P \boldsymbol{b}$ and $P^2=P=P^{\mathrm{T}}$.

enter image description here

$\endgroup$
36
  • $\begingroup$ Comments have been moved to chat; please do not continue the discussion here. Before posting a comment below this one, please review the purposes of comments. Comments that do not request clarification or suggest improvements usually belong as an answer, on Mathematics Meta, or in Mathematics Chat. Comments continuing discussion may be removed. $\endgroup$
    – Xander Henderson
    Apr 19 at 1:56
  • 1
    $\begingroup$ @JonathanZ Updated. Thanks! $\endgroup$ Apr 19 at 2:31
  • $\begingroup$ @SamKirkiles ""induced by the inner product (usually the $L^2$ norm).""-- what is L2 here? what does meaning of $L^2$ norm? $\endgroup$
    – David
    Apr 19 at 2:47
  • $\begingroup$ The standard euclidean norm $\|v\|_2 = \sqrt{\sum_{i}x_i^2}$. If you haven't seen different norms its not important. It doesn't matter in finite dimensions. Basically we can't do orthogonal projections unless we are in an inner product space. But in finite dimensions, we can always get an inner product no problem. out of all the norms $\|\cdot\|_p$ $p=2$ is the only one that makes an inner product. It's called a hilbert space. $\endgroup$ Apr 19 at 2:48
  • $\begingroup$ What do you mean holding three points? $\endgroup$ Apr 19 at 2:54
4
$\begingroup$

The theorem you cite is very general, and your questions are very basic, so I think it might be helpful for you to first see how the theorem plays out in a very simple situation. (I'm going to write $P_i$ instead of $p_i$ - after all, they are linear transformations/matrices just like $A$ is.)

Okay, let's let

$$A=\left(\begin{array}{cc} -2 & 0 & 0 & 0 \\ 0 & 3 & 0 & 0 \\ 0 & 0 & 3 & 0 \\ 0 & 0 & 0 & 5 \\ \end{array}\right)$$

Then it's easy to see that

$$ A = -2\left(\begin{array}{cc} 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ \end{array}\right) +3\left(\begin{array}{cc} 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 \\ \end{array}\right) +5\left(\begin{array}{cc} 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ \end{array}\right) $$

and this sum corresponds to the expression $\sum{\lambda_i P_i}$ in your statement: the $\lambda_i$ are $-2, 3$, and $5$, and the $P_i$ are the matrices.

As for your three points:

1 - You should already know what a projection is, what it means for a projection to be orthogonal, and what it means to project onto a subspace. These are all prerequisites to this theorem, and if you don't know what they mean you need to go read up on them, or ask a separate question.

2 - You've got a mistake there, as it should say "$P_i P_j = 0$ if $i\ne j$". It's easy enough to verify that this is true in this case; just do the matrix multiplications, they're mostly zeros. And what it means, roughly speaking, is that each $P_i$ projects on to a different subspace, and there is no "overlap" between the subspaces.

3 - If you apply $I = P_1 + P_2 + P_3$ to some vector $v$, it means that

$$ v = P_1 v + P_2 v + P_2 v,$$

i.e. it allows you to write any vector as a sum of other vectors - and not just any vectors, but eigenvectors. One could say that it resolves it into eigenvectors, which is why the other answer described this as "a resolution of the identity".

To take a concrete example, suppose $v = \left(\begin{array}{cc} 8 \\ 2 \\ 5 \\ 1 \\ \end{array}\right)$. Applying the resolution of the identity here, we get

$$v = \left(\begin{array}{cc} 8 \\ 0 \\ 0 \\ 0 \\ \end{array}\right)+ \left(\begin{array}{cc} 0 \\ 2 \\ 5 \\ 0 \\ \end{array}\right)+ \left(\begin{array}{cc} 0 \\ 0 \\ 0 \\ 1 \\ \end{array}\right).$$

Now, I'm expecting that you know and understand that the same linear transformation will be represented by a different matrix of numbers, depending on the basis one chooses to uses. This particular $A$ had the standard unit vectors as its basis of eigenvectors, so everything looks simple. For other transformations the eigenvectors won't be "mostly zeros", so the matrices will be more complicated, but all of the equations you've listed will still hold true.

$\endgroup$
7
  • 1
    $\begingroup$ Would you write short easy note on projection and orthogonal projection by small example, and when projection to be orthogonal? $\endgroup$
    – David
    Apr 19 at 1:53
  • 2
    $\begingroup$ Sorry, @Brett, but those are basic things. You should at least first go read the Wikipedia article(s) covering them, (or the parts of your text that cover them), and post questions about the parts that you don't understand. I'm not going to just repeat all that material here. $\endgroup$
    – JonathanZ
    Apr 19 at 1:57
  • 1
    $\begingroup$ I understand everything, but how orthogonal projection connects with eigen space, I need to know. Would you please elaborate this at least. $\endgroup$
    – David
    Apr 19 at 1:59
  • 3
    $\begingroup$ Any time you have a subspace you can find an operator/matrix that projects onto it. In fact you can find lots of them - they'll all have the same range, but they will have different null spaces. The orthogonal projection is the one of these projection for which the null space is the orthogonal complement of the range. As for the connection with eigenspaces, well, eigenspaces are subspaces that help us understand a linear transformation, so having the orthog. projection for it gives us an operator/matrix that tells us about that eigenspace. $\endgroup$
    – JonathanZ
    Apr 19 at 2:06
  • 1
    $\begingroup$ @JonathanZ would you cite me any website link in where I can easily understand projection and orthogonal projection of matrix. $\endgroup$
    – David
    Apr 19 at 13:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .