# Intuition behind $x^T A x$

I have been encountering the expression $$x^T A x$$ (where $$x$$ is a vector and $$A$$ is a matrix) in many topics related to linear algebra. However, I fail to grasp the essence; what is happening here? For example, I know (or more like, have been told) that if the expression is greater than zero for all vectors $$x$$, then we claim that $$A$$ is positive-definite. I would like to understand why such property - and many others - matter by understanding why should we care about $$x^T A x$$.

I understand that $$x^T x$$ is a dot product and that $$x^T A$$ and $$A x$$ are vectors of linear combinations of $$x$$ defined by columns and rows of $$A$$ respectively.

• It's just a high dimensional parabola like $ax^2$. Being positive definite or negative definite translates to it being "smiling" or "frowning". Sometimes though, its neither. Oct 20, 2020 at 10:33

1. A first intuitive answer is by comparison between first degree monomial $$ax$$ which, rather faithfully, is reflected as $$AX$$ ; if we want something analogous for second degree monomial $$ax^2$$, we cannot write $$AXX$$ because dimensions do not match. It has been noticed that $$X^TAX$$ is valid from the point of view of dimensions, and also

a) from the point of view of algebraic developments like

$$(X_1+X_2)^TA(X_1+X_2)=X_1^TAX_1+2X_1^TAX_2+X_2^TAX_2 \tag{1}$$

(due to the fact that $$X_1^TAX_2=X_2^TAX_1$$ because one is the transpose of the other and that the transpose of a number is the number itself), and (1) is in full correspondence with one of the most basic algebraic identities: $$a(x_1+x_2)^2=ax_1^2+2ax_1x_2+ax_2^2$$

b) A second reason is that, for differential expressions, we have a perfect equivalent of the derivative of $$ax^2$$ is $$2ax$$. Indeed (1) can be written:

$$\underbrace{(X+H)^TA(X+H)}_{q(X+H)}=\underbrace{X^TAX}_{q(X)}+H^T\underbrace{2AX}_{q'(X)}+H^TAH \tag{2}$$

(think to $$H$$ as a small vector increment) where we recognize a Taylor expansion with the derivative at it's right place.

2- A second category of intuitive explanation is that this notation is perfectly adapted to the description conic curves. For example, ellipse with equation $$\dfrac{x^2}{a^2}+\dfrac{y^2}{b^2}=1$$ will be written $$X^TAX=\begin{pmatrix}x & y & 1\end{pmatrix}\begin{pmatrix}\dfrac{1}{a^2} &0&0\\0&\dfrac{1}{b^2}&0\\0&0&-1 \end{pmatrix}\begin{pmatrix}x \\ y \\ 1\end{pmatrix}.$$ A lot of information can be extracted from this expression: its eigenvalues, its inverse, certain determinants, have a geometric signification.

3 - Last but not least, the duality brought by bilinear form :

$$b(X,Y)=X^TAY$$

with a generalized orthogonality

$$b(X,Y)=Y^TAX=0\tag{2}$$

introducing an dual (assymetric...) connection between a line associated with $$X$$ and a point $$Y$$=(x_0,y_0,1)\$ [the twinned pair is called a pole-polar pair]. Let us illustrate it on the case of the ellipse for different situations: points outside and on the ellipse (in the last case, we get the tangent)

If the pole is $$X_0=\begin{pmatrix}x_0\\y_0\\1 \end{pmatrix}$$, the set of vectors $$Y=\begin{pmatrix}x\\y\\1\end{pmatrix}$$ such that $$Y^TAX=0$$ is a line (the polar line of $$X_0$$) with equation

$$xx_0/a^2+yy_0/b^2-1=0$$

Four of these pole-polar pairs are represented on the figure below, each one with a specific color. The green one illustrates the fact that pole-polar relationship covers (said otherwise generalizes) the case of point of the curve with its associated tangent line.

Here are some reasons to care about $$x^tAx$$.

1. Let $$f: \mathbb{R}^n \rightarrow \mathbb{R}$$ be $$C^2$$. Then $$f(x+h)=f(x)+\nabla f(x)\cdot h+h^tH_f(\xi)h$$ for some $$\xi$$ on the segment from $$x$$ to $$x+h$$ (and $$H_f(\xi)$$ is the Hessian of $$f$$ at $$\xi$$). So you can regard this product as a "second order" contribute to approximating $$f$$ at $$x$$. Furthermore if $$H_f(x)$$ is positive definite $$f$$ has a strict local minimum at $$x$$

2. If $$A$$ is positive definite, then the function $$x \rightarrow x^tAx$$ is strictly convex, and you have at most one global minimizer.

To me most obvious use of $$x^TAx$$ is that it can represent lot's of quadratic functions.

For example, if: $$x=\begin{pmatrix} x \\ y \\ \end{pmatrix}$$ and $$A=\begin{pmatrix} a & b \\ b & c \\ \end{pmatrix}$$ the expression will result in: $$x^TAx=ax^2 + byx + cy^2$$

So by choosing the right values for $$(a,b,c)$$ you can encode a variety of quadratics. Whether the matrix $$A$$ is positive-definite matters a lot in optimization problems, because if the matrix is positive-definite the resulting function will be convex, and will have a minimum.

You can improve on the reach of quadratic functions you can represent by adding the terms $$x^Tb + c$$ where $$b$$ is a n-vector and $$c$$ is a real a constant. This way you have $$x^TAx + x^Tb + c$$, which can represent any quadratic. And again, it will have a minimum vector, if and only if the matrix $$A$$ is positive-definite.

I hope you have gained some insight as to why the properties of the matrix $$A$$ are important and why $$x^TAx$$ is too. Am happy to answer any doubts about what I've written.