# Intuitive explanation of the Fundamental Theorem of Linear Algebra

Can someone explain intuitively what the Fundamental Theorem of Linear Algebra states? and why specifically it is called the above? Specifically, what makes it 'Fundamental' in the broad scope of the theory.

-
Read en.wikipedia.org/wiki/Fundamental_theorem_of_linear_algebra and the article by Strang in the Monthly cited in that page. –  lhf Feb 9 '11 at 9:40
Argh, I rather disliked this "fundamental theorem". The whole statement in terms of orthogonality of the kernel and the co-image is obscuring the real statement about vector spaces and their duals. The fundamentality of the theorem (the "non-obvious" relation between kernel, cokernel, image, and coimage) is only due to the setting of working on $\mathbb{R}^n$ and identifying with its dual space using the standard inner product. (The other half of the theorem [rank-nullity], however, is a very fundamental one, I think. It makes the definition of dimension sensible.) –  Willie Wong Feb 9 '11 at 18:03
It all boils down to this: The only invariant of a finitely generated vector space is its dimension. –  Christian Blatter May 17 '11 at 17:28
@littleO: the statement is relatively trivial. Let $T:V\to W$ be a linear map, it induces the dual map $T^*: W^* \to V^*$ where $W^*$ is the dual space of $W$. Then by definition the duality pairing $\langle T^* w^*,v\rangle = \langle w^*,Tv\rangle = 0$ for every $w^*\in W^*$ and for every $v$ such that $Tv = 0$. Or in words, "the image of $W^*$ under $T^*$ annihilates all members of the kernel of $T$". –  Willie Wong Aug 14 '13 at 9:08
Never heard of a theorem called like that. At first sight the result is even more localised than the Fundamental Theorem of Algebra. Nothing for other fields than $\Bbb R$, nothing for abstract spaces and morphisms, nothing for infinite dimensional spaces. Just curious, is there any indication of anybody using this nomenclature independently of Gilbert Strang? –  Marc van Leeuwen Aug 14 '13 at 10:10

Imagine a projection, for example, from the whole $\mathbb{R}^3$ to the $x$-$y$ plane. It compress each line that parallels to $z$-axis to a point on the plane. So there is a one-to-one relationship between the lines and the points. Notice that the lines are the translations of the $z$-axis -- that is also what quotient means. And the $z$-axis, is just the kernel of the projection, so we can see that $\operatorname{im} A\simeq V/\ker A$.

As for the dimension, the dimension of $\ker A$ measures how much we compress, while the dimension of $\operatorname{im} A$ measures how much we leave -- the amount we compress, plus the amount we leave, equals the whole thing, intuitively.

-

The theorem is fundamental because it rests upon two elementary ideas for solving for vector $x$ in a linear homogeneous algebraic equation $Ax=0$.

The fundamental theorem of linear algebra (FTLA) has two parts, each originating from simple ideas in college algebra, especially the topic of linear algebraic equations in the case of infinitely many solutions.

A quick example of a $3 \times 3$ homogeneous linear algebraic system in scalar form:

$$\begin{cases} &x + 2y + 3z = 0, \\ &0=0, \\ &0=0. \\ \end{cases}$$

The system has infinitely many solutions, expressed in terms of the lead variable $x$ and free variables $y,z$. The general scalar solution is

\begin{align} x &= - 2t - 3s, \\ y &= t, \\ z &= s. \\ \end{align}

where $t,s$ are invented symbols for the free variables.

IDEA 1: In the equation $Ax=0$, the number, $n$, of components in vector $x$ equals the lead variable count plus the free variable count.

IDEA 2: Any solution $x$ of $Ax=0$ is perpendicular to every row of $A$.

Idea 1 is called the rank-nullity theorem, part 1 of the FTLA. The rank is the lead variable count and the nullity is the free variable count.

Idea 2 is part 2 of the FTLA, presented in textbooks as a deeper result using the notation of kernel, image and orthogonal complement. The FTLA part 2 is opaque. But Idea 2 is not itself opaque: it is visible from the definition of matrix times vector (the meaning of $Ax$). In short, you can derive the result in Idea 2 by looking at the equation $Ax=0$, almost no background required.

-
+1 Excellent answer... welcome to Math.SE! I made some formatting changes including writing the mathematics in $\LaTeX$. You can see the MathJax basic tutorial and quick reference to learn how to do this. If you would like to modify or improve your answer, you may do so by pressing the 'edit' button at the bottom left of the post. –  Gamma Function Aug 14 '13 at 8:44

Suppose $V$ and $W$ are finite dimensional inner product spaces over $F$ (where $F$ is $\mathbb R$ or $\mathbb C$). Let $T:V \to W$ be a linear transformation.

There are four subspaces that are naturally associated with $T$: $N(T), R(T), N(T^*)$, and $R(T^*)$. (Here $T^*$ is the adjoint of $T$. So $\langle T(x), y \rangle = \langle x, T^*(y) \rangle$ for all $x \in V, y \in W$.)

What Strang calls the fundamental theorem of Linear Algebra is the fact that $$V = N(T) \perp R(T^*)$$ and $$W = N(T^*) \perp R(T).$$

This theorem is easy to prove: \begin{align} & x \in N(T) \\ \iff & T(x) = 0 \\ \iff & \langle T(x),y \rangle = 0 \forall y \in W \\ \iff & \langle x, T^*(y) \rangle = 0 \forall y \in W \\ \iff & x \in R(T^*)^{\perp}. \end{align} This shows that $N(T)$ is the orthogonal complement of $R(T^*)$. Similarly, $N(T^*)$ is the orthogonal complement of $R(T)$.

It's natural to seek bases for these four subspaces, and perhaps the "most natural" or nicest bases are the ones that arise in the SVD of $T$.

Edit: It's interesting that, as @WillieWong pointed out, a similar theorem can be formulated in a more general setting, without using inner product spaces.

Let $V$ and $W$ be finite dimensional vector spaces over a field $F$ and let $T:V \to W$. There are four subspaces naturally associated with $T$: $N(T), R(T), N(T^*)$, and $R(T^*)$. (Now $T^*$ is the dual transformation $T^*:W^* \to V^*$.)

We may not have an inner product to work with, but we can still write $$\langle T^*(w^*),v \rangle = \langle w^*,T(v) \rangle$$ if we use the notation $$\langle x^*,y \rangle := x^*(y)$$ for $x^* \in V^*, y \in V$.

Moreover, $N(T)$ may not have an orthogonal complement, but it does have an annihilator, which is the analogous thing in a more general setting. And we have that $R(T^*)$ is the annihilator of $N(T)$, and also that $N(T^*)$ is the annihilator of $R(T)$. This is a more general form of Strang's "fundamental theorem".

Note that for any subspace $U$ of $V$, we have $$\text{dim} \, U + \text{dim} \,U^{\perp} = \text{dim} \, V$$ where $U^{\perp}$ is the annihilator of $U$. In particular, $$\text{dim} \, N(T) + \text{dim} \, R(T^*) = \text{dim} \, V.$$ But we also know that $$\text{dim} \, N(T) + \text{dim} \, R(T) = \text{dim} \, V.$$ This shows that $$\text{dim} \, R(T) = \text{dim} \, R(T^*).$$

These easy but fundamental theorems and proofs appear in chapter 2 of Lax's book Linear Algebra and its Applications. (See theorems 5, 5', and 6.)

-