What is the intuition for using definiteness to compare matrices?

Question

If $a$ and $b$ are two numbers on the real line, we compare $a$ and $b$ by knowing which of them comes first as we move from $-\infty$ to $\infty$ on the real line.

However when $A$ and $B$ are matrices, the comparison is through definiteness. We say $A \succ B$ iff $A-B$ is positive definite. Positive definiteness of $A$ means $x^TAx>0\ \forall x$; essentially the function $f(x)=x^TAx$ takes the form of a bowl with its base at origin.

How does this "bowl" help in comparing two matrices? What is the intuition behind using definiteness in matrices for ordering?

Answer 1

Where did you get your definition from?

I think you might be confusing the notation here. $A \succ B$ simply is a notation used to indicate that $A$ and $B$ are symmetric matrices of same size such that $A - B$ is positive definite. The reason such a notation is used is because there are certain (monotonic) properties these matrices satisfy. For example, if $A \succ B$ then $\lambda(A) > \lambda(B)$.

Now, there is another similar relation which says $A \succeq B$ which is used to express that $A$ and $B$ are symmetric matrices of same size such that $A - B$ is positive semi-definite. Now, this also has similar (monotonic) properties as the relation above.

However, one thing about the $\succeq$ relation is that it does define a partial order over all symmetric matrices (You can take this as an exercise, it is quite easy to show). This is called Lowner ordering. In this sense, one can say that we are comparing $A$ with $B$. But again, you have to understand that it is not for arbitrary matrices, only for symmetric matrices. Also, this ordering is partial and not total.

Answer 2

The comparison relation you describe only really makes sense for symmetric matrices, since $x^TSx=0$ for every skew-symmetric matrix $S$, and therefore this relation would treat any matrix as equal to its symmetrisation. Symmetric matrices correspond to quadratic forms by associating to $A$ the quadradic form $x\mapsto x^TAx$ (one can recover the matrix from the quadratic form), and the relation $A\succ B$ means exactly that the quadradic form associated to $A$ takes everywhere strictly larger values than the quadradic form associated to $B$, that is $x^TAx>x^TBx$ for all $x$ (except for $x=0$, which you should have excluded in your definition). When this is the case, it is quite natural to consider that $A$ dominates $B$. If one defines $A\succeq B$ by $x^TAx\geq x^TBx$ for all $x$ (warning: this includes cases where neither $A\succ B$ nor $A=B$ holds), then this defines a partial ordering on symmetric matrices. The "partial" is because there are many pairs $A,B$ where neither $A\succeq B$ nor $A\preceq B$ holds: the quadratic form for $A$ takes values larger than that for $B$ in some vectors, and smaller values for other vectors.

Answer 3

We're not actually interested in the behavior of the graph of the function $f(x) = x^T Ax$, other than the sign of this function. Consider the case of $a,b$ as two numbers on the real line. Then they can be thought of as $1 \times 1$ matrices which act on $\mathbb{R}^1$ vectors. Then you can determine whether $a$ and $b$ are positive, respectively, by looking at the functions $f(x) = ax^2$ and $g(x) = bx^2$ and see whether the parabolas are pointing upward or downward (or, as an old school teacher of mine used to say, happy or sad). We are not actually concerned with the specific values that these functions take, other than to inspect their sign.

Bumping this up to $n$-dimensions, the map $f(x) = x^T A x$ is a quadratic form $\mathbb{R}^n \rightarrow \mathbb{R}$, and, as you said, their graphs look like "bowls" in $n$-dimensional space. Every cross-section of this graph looks like a "happy" parabola. If one of the eigenvalues of $A$ is negative instead of positive, then some cross-sections of the graph will be "sad".

Answer 4

Suppose $A$ and $B$ symmetric and positive definite. Positive definite matrices are diagonalizable and are equivalent (ie conjugated) to a positive definite diagonal matrix. More precidamente we have the following.

We have that $A\succ B$ if, only if, all eigenvalues of $A-B$ are positives and distinct i.e. $\lambda_1(A-B)=u_1^T(A-B)u_1>\lambda_2(A-B)=u_2^T(A-B)u_2>\dots >\lambda_{n-1}(A-B)=u_{n-1}^T(A-B)u_{n-1}>\lambda_n(A-B)=u_n^T(A-B)u_n>0$. Here $u_1,\dots,u_n$ are the eigenvectors unitary of $(A-B)$.

Therefore an intuitive way to compare two matrix $A,B$ in this partial order is equivalent to compare the eigenvalues ​​of the diagonal matrix of $A-B$ whit $0$.

Answer 5

This is a way to compare quadratic forms, more than matrices. $A\prec B$ means exactly that the quadratic form associated to $B$ dominates pointwise the one associated to $A$. Moreover, this explains why you can only order this way symmetric matrices: they are the ones who correspond bijectively to quadratic forms.