4
$\begingroup$

I know the matrix differentiation of $x^TAx$ is if A is symmetric is 2Ax. I saw that some people write as $2x^TA$. Are these two results the same?

I am not sure how they are same. Can anyone explain to me..

$\endgroup$
1
  • $\begingroup$ You should clarify that you want the matrix derivative wrt $x$ $\endgroup$ Commented Apr 24, 2020 at 22:57

4 Answers 4

4
$\begingroup$

The function $f:\mathbb{R}^{n}\to\mathbb{R}$ defined by $f(x)=x^{T}Ax$ should have it's Jacobian matrix, $f'(x)$, be of size $1\times n$. So the appropriate form of the Jacobian matrix should be $2x^{T}A$. They are not equal as matrices in general (one is a row vector and one is a column vector). However, $(2x^{T}A)^{T}=2A^{T}(x^{T})^{T}=2Ax$ since $A$ is symmetric. The last sentence shows that they differ only by a matrix transposition. It may be that the person who wrote that was using the convention that $\nabla f(x)=f'(x)^{T}$ in which case they were referring to the $\nabla f(x)$ instead of the Jacobian matrix.

$\endgroup$
2
  • 1
    $\begingroup$ Note that, while $f'(x)$ is a $1 \times n$ matrix, some people use the convention that $\nabla f(x)$ is an $n \times 1$ column vector. So by that convention we would have $\nabla f(x) = f'(x)^T$. For example, in optimization literature $\nabla f(x)$ is usually a column vector. $\endgroup$
    – littleO
    Commented Sep 6, 2013 at 2:43
  • $\begingroup$ I was not aware of that convention. Thank you for the information littleO. $\endgroup$
    – user71352
    Commented Sep 6, 2013 at 2:46
2
$\begingroup$

Let $\phi_A$ denote the map taking $x \to x^TAx$, so that $\phi_A(x) = x^TAx$. Just for the record, let us suppose $x \in V$ for some vector space $V$, with field of scalars $\Bbb F$, where $\Bbb F = \Bbb R$ or $\Bbb F = \Bbb C$, the real or complex numbers. Then, for the sake of definiteness, we can say $\phi_A: V \to \Bbb F$. Now if we look at $\phi_A(x + h)$ we see that

$\phi_A(x + h) = (x + h)^TA(x + h) = x^TAx + h^TAx + x^TAh + h^TAh, \tag{1}$

or

$\phi_A(x + h) = \phi_A(x) + h^TAx + x^TAh + \Vert h \Vert o(h), \tag{2}$

where $o(h)$ is a continuous function such that $\lim_{h \to 0} o(h) = 0$. The derivative $D_x\phi_A$ of $\phi_A$ at $x$ is thus the linear functional $D_x\phi_A:V \to \Bbb F$ given by the formula

$D_x\phi_A(h) = h^TAx + x^TAh, \tag{3}$

which holds for all $A$, symmetric or not. Now since $h^TAx \in \Bbb F$ is a scalar it is "automatically" symmetric; it is equal to its own transpose: $h^TAx = (h^TAx)^T = x^TA^Th^{TT} = x^TA^Th$, whence

$D_x\phi_A(h) = x^TA^Th + x^TAh, \tag{4}$

and a similar maneuver may be applied to $x^TAh$: $x^TAh = (x^TAh)^T = h^TA^Tx^{TT} = h^TA^Tx$, yielding an alternative form of $D_x\phi_A(h)$:

$D_x\phi_A(h) = h^TAx + h^TA^Tx, \tag{5}$

(4) and (5) are about as far as you can go without additional assumptions on $A$; in the event $A$ is symmetric, i.e. $A = A^T$, (4) and (5) become

$D_x\phi_A(h) = 2x^TAh, \tag{6}$

and

$D_x\phi_A(h) = 2h^TAx, \tag{7}$

respectively. I believe this shows your two formula are about as "the same" as they can get, modulo the exact choice of which vector to transpose; but you've got to do it someplace!

Hope this helps. Cheers.

$\endgroup$
2
$\begingroup$

The gradient of a quadratic form $x^T A x$ is given by

$\nabla x^T A x = (A^T + A)x$

which is valid for arbitrary (non-symmetric) $A$. If $A$ is symmetric of course $2 A x$ is obtained. If in this case ($A$ being symmetric) the gradient is written as row vector the other form $2 x^T A$ is obtained.

$\endgroup$
0
$\begingroup$

If you view the elements of $\mathbb{R}^n$ as column vectors, then the derivative at $x_0$ of the function $$f : \mathbb{R}^n \to \mathbb{R}, \quad x \mapsto x^T A x,$$ is the linear functional $$Df(x_0) : \mathbb{R}^n \to \mathbb{R}, \quad h \mapsto x_0^T(A+A^T)h,$$ which is represented by the $1 \times n$ matrix (i.e., row vector) $$ x_0^T(A+A^T). $$ On the other hand, by definition, the gradient at $x_0$ of $f$ is the column vector $\nabla f(x_0)$ such that for all $h \in \mathbb{R}^n$, $$ Df(x_0)h = (\nabla f(x_0))^T h, $$ so that $$ (\nabla f(x_0))^T h = Df(x_0)h = x_0^T(A+A^T)h = ((A+A^T)x_0)^T h $$ for all $h$, and hence $$ \nabla f(x_0) = (A+A^T)x_0. $$

$\endgroup$

Not the answer you're looking for? Browse other questions tagged .