2
$\begingroup$

I have been encountering the expression $x^T A x$ (where $x$ is a vector and $A$ is a matrix) in many topics related to linear algebra. However, I fail to grasp the essence; what is happening here? For example, I know (or more like, have been told) that if the expression is greater than zero for all vectors $x$, then we claim that $A$ is positive-definite. I would like to understand why such property - and many others - matter by understanding why should we care about $x^T A x$.

I understand that $x^T x$ is a dot product and that $x^T A$ and $A x$ are vectors of linear combinations of $x$ defined by columns and rows of $A$ respectively.

$\endgroup$
1
  • $\begingroup$ It's just a high dimensional parabola like $ax^2$. Being positive definite or negative definite translates to it being "smiling" or "frowning". Sometimes though, its neither. $\endgroup$
    – Theorem
    Oct 20, 2020 at 10:33

3 Answers 3

2
$\begingroup$

There are different ways to answer your question.

  1. A first intuitive answer is by comparison between first degree monomial $ax$ which, rather faithfully, is reflected as $AX$ ; if we want something analogous for second degree monomial $ax^2$, we cannot write $AXX$ because dimensions do not match. It has been noticed that $X^TAX$ is valid from the point of view of dimensions, and also

a) from the point of view of algebraic developments like

$$(X_1+X_2)^TA(X_1+X_2)=X_1^TAX_1+2X_1^TAX_2+X_2^TAX_2 \tag{1}$$

(due to the fact that $X_1^TAX_2=X_2^TAX_1$ because one is the transpose of the other and that the transpose of a number is the number itself), and (1) is in full correspondence with one of the most basic algebraic identities: $$a(x_1+x_2)^2=ax_1^2+2ax_1x_2+ax_2^2$$

b) A second reason is that, for differential expressions, we have a perfect equivalent of the derivative of $ax^2$ is $2ax$. Indeed (1) can be written:

$$\underbrace{(X+H)^TA(X+H)}_{q(X+H)}=\underbrace{X^TAX}_{q(X)}+H^T\underbrace{2AX}_{q'(X)}+H^TAH \tag{2}$$

(think to $H$ as a small vector increment) where we recognize a Taylor expansion with the derivative at it's right place.

2- A second category of intuitive explanation is that this notation is perfectly adapted to the description conic curves. For example, ellipse with equation $\dfrac{x^2}{a^2}+\dfrac{y^2}{b^2}=1$ will be written $$X^TAX=\begin{pmatrix}x & y & 1\end{pmatrix}\begin{pmatrix}\dfrac{1}{a^2} &0&0\\0&\dfrac{1}{b^2}&0\\0&0&-1 \end{pmatrix}\begin{pmatrix}x \\ y \\ 1\end{pmatrix}.$$ A lot of information can be extracted from this expression: its eigenvalues, its inverse, certain determinants, have a geometric signification.

3 - Last but not least, the duality brought by bilinear form :

$$b(X,Y)=X^TAY$$

with a generalized orthogonality

$$b(X,Y)=Y^TAX=0\tag{2}$$

introducing an dual (assymetric...) connection between a line associated with $X$ and a point $Y$=(x_0,y_0,1)$ [the twinned pair is called a pole-polar pair]. Let us illustrate it on the case of the ellipse for different situations: points outside and on the ellipse (in the last case, we get the tangent)

If the pole is $X_0=\begin{pmatrix}x_0\\y_0\\1 \end{pmatrix}$, the set of vectors $Y=\begin{pmatrix}x\\y\\1\end{pmatrix}$ such that $Y^TAX=0$ is a line (the polar line of $X_0$) with equation

$$xx_0/a^2+yy_0/b^2-1=0$$

Four of these pole-polar pairs are represented on the figure below, each one with a specific color. The green one illustrates the fact that pole-polar relationship covers (said otherwise generalizes) the case of point of the curve with its associated tangent line.

enter image description here

$\endgroup$
1
$\begingroup$

Here are some reasons to care about $x^tAx$.

  1. Let $f: \mathbb{R}^n \rightarrow \mathbb{R}$ be $C^2$. Then $f(x+h)=f(x)+\nabla f(x)\cdot h+h^tH_f(\xi)h$ for some $\xi$ on the segment from $x$ to $x+h$ (and $H_f(\xi)$ is the Hessian of $f$ at $\xi$). So you can regard this product as a "second order" contribute to approximating $f$ at $x$. Furthermore if $H_f(x)$ is positive definite $f$ has a strict local minimum at $x$

  2. If $A$ is positive definite, then the function $x \rightarrow x^tAx$ is strictly convex, and you have at most one global minimizer.

$\endgroup$
1
$\begingroup$

To me most obvious use of $x^TAx$ is that it can represent lot's of quadratic functions.

For example, if: $x=\begin{pmatrix} x \\ y \\ \end{pmatrix}$ and $A=\begin{pmatrix} a & b \\ b & c \\ \end{pmatrix}$ the expression will result in: $$x^TAx=ax^2 + byx + cy^2$$

So by choosing the right values for $(a,b,c)$ you can encode a variety of quadratics. Whether the matrix $A$ is positive-definite matters a lot in optimization problems, because if the matrix is positive-definite the resulting function will be convex, and will have a minimum.

You can improve on the reach of quadratic functions you can represent by adding the terms $x^Tb + c$ where $b$ is a n-vector and $c$ is a real a constant. This way you have $x^TAx + x^Tb + c$, which can represent any quadratic. And again, it will have a minimum vector, if and only if the matrix $A$ is positive-definite.

I hope you have gained some insight as to why the properties of the matrix $A$ are important and why $x^TAx$ is too. Am happy to answer any doubts about what I've written.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .