41
$\begingroup$

For the Quadratic Form $X^TAX; X\in\mathbb{R}^n, A\in\mathbb{R}^{n \times n}$ (which simplifies to $\Sigma_{i=0}^n\Sigma_{j=0}^nA_{ij}x_ix_j$), I tried to take the derivative wrt. X ($\Delta_X X^TAX$) and ended up with the following:

The $k^{th}$ element of the derivative represented as

$\Delta_{X_k}X^TAX=[\Sigma_{i=1}^n(A_{ik}x_k+A_{ki})x_i] + A_{kk}x_k(1-x_k)$

Does this result look right? Is there an alternative form?

I'm trying to get to the $\mu_0$ of Gaussian Discriminant Analysis by maximizing the log likelihood and I need to take the derivative of a Quadratic form. Either the result I mentioned above is wrong (shouldn't be because I went over my arithmetic several times) or the form I arrived at above is not the terribly useful to my problem (because I'm unable to proceed).

I can give more details about the problem or the steps I put down to arrive at the above result, but I didn't want to clutter to start off. Please let me know if more details are necessary.

Any link to related material is also much appreciated.

$\endgroup$
0

5 Answers 5

70
$\begingroup$

Let $Q(x) = x^T A x$. Then expanding $Q(x+h)-Q(x)$ and dropping the higher order term, we get $DQ(x)(h) = x^TAh+h^TAx = x^TAh+x^TA^Th = x^T(A+A^T)h$, or more typically, $\frac{\partial Q(x)}{\partial x} = x^T(A+A^T)$.

Notice that the derivative with respect to a column vector is a row vector!

$\endgroup$
24
  • 2
    $\begingroup$ What do you mean? Just compute $Q(x+h)-Q(x)$ explicitly. The only term missing above is $h^T A h$, and we have $|h^T A h| \le \|A\| \|h \|^2$, so the term is $O(\|h\|^2)$. $\endgroup$
    – copper.hat
    Feb 6, 2014 at 21:15
  • 6
    $\begingroup$ There is no need to explicitly compute the sums. Matrix multiplication is associative and distributive, so we can treat them like 'numbers' in this regard. We have $A(x+h) = Ax + Ah$, $(x+h)^TA = (x^T +h^T) A = x^TA + h^T A$, etc. $\endgroup$
    – copper.hat
    Feb 7, 2014 at 16:22
  • 2
    $\begingroup$ @Sother: Cauchy Schwarz gives $|\langle h, Ah \rangle | \le \|h\| \|Ah\|$ and (if we use the Euclidean norm) we have $\|Ah\| \le \|A\| \|h\|$. $\endgroup$
    – copper.hat
    Sep 12, 2017 at 15:16
  • 2
    $\begingroup$ I just learned a new trick when your independent variable is in more than two places within your formula: introduce a new (fake) parameter which will then disappear: $$\frac{\partial}{\partial x} y^TAx = \frac{\partial y}{\partial x}[Ax]^T+y^TA $$ The transpose was to make the vector a row vector. Nothing deep there! Now, if $y=x$ then $$ \frac{d}{dx} x^TAx = x^TA^T+x^TA = x^T(A+A^T) \ . $$ $\endgroup$ Sep 18, 2017 at 21:38
  • 2
    $\begingroup$ @AnkurRoyChowdhury: I don't understand your question. $h$ is a vector, you can't divide by $h$. I didn't divide by $h$ anywhere. Note that the map $DQ(x)$ is a map $\mathbb{R}^n \to \mathbb{R}^n$, which can be represented by a matrix ${\partial Q(x) \over \partial x}$. In other words, $DQ(x)h = {\partial Q(x) \over \partial x}h$. $\endgroup$
    – copper.hat
    Mar 16, 2019 at 20:39
16
$\begingroup$

You could also take the derivative of the scalar sum. \begin{equation} \begin{aligned} {\bf x^TAx} = \sum\limits_{j=1}^{n}x_j\sum\limits_{i=1}^{n}x_iA_{ji} \end{aligned} \end{equation} The derivative with respect to the $k$-th variable is then(product rule): \begin{equation} \begin{aligned} \frac{d {\bf x^TAx}}{d x_k} & = \sum\limits_{j=1}^{n}\frac{dx_j}{dx_k}\sum\limits_{i=1}^{n}x_iA_{ji} + \sum\limits_{j=1}^{n}x_j\sum\limits_{i=1}^{n} \frac{dx_i}{dx_k}A_{ji} \\ & = \sum\limits_{i=1}^{n}x_iA_{ki} + \sum\limits_{j=1}^{n}x_jA_{jk} \end{aligned} \end{equation}

If then you arrange these derivatives into a column vector, you get: \begin{equation} \begin{aligned} \begin{bmatrix} \sum\limits_{i=1}^{n}x_iA_{1i} + \sum\limits_{j=1}^{n}x_jA_{j1} \\ \sum\limits_{i=1}^{n}x_iA_{2i} + \sum\limits_{j=1}^{n}x_jA_{j2} \\ \vdots \\ \sum\limits_{i=1}^{n}x_iA_{ni} + \sum\limits_{j=1}^{n}x_jA_{jn} \\ \end{bmatrix} = {\bf Ax} + ({\bf x}^T{\bf A})^T = ({\bf A} + {\bf A}^T){\bf x} \end{aligned} \end{equation}

or if you choose to arrange them in a row, then you get: \begin{equation} \begin{aligned} \begin{bmatrix} \sum\limits_{i=1}^{n}x_iA_{1i} + \sum\limits_{j=1}^{n}x_jA_{j1} & \sum\limits_{i=1}^{n}x_iA_{2i} + \sum\limits_{j=1}^{n}x_jA_{j2} & \dots & \sum\limits_{i=1}^{n}x_iA_{ni} + \sum\limits_{j=1}^{n}x_jA_{jn} \end{bmatrix} \\ = ({\bf Ax} + ({\bf x}^T{\bf A})^T)^T = (({\bf A} + {\bf A}^T){\bf x})^T = {\bf x}^T({\bf A} + {\bf A}^T) \end{aligned} \end{equation}


$\endgroup$
6
$\begingroup$

It is easier using index notation with Einstein (repeated sum on dummy indices) rule. That is, we can write the $i$th component of $Ax$ as $a_{ij} x_j$, and $x^T A x=x_i a_{ij} x_j = a_{ij} x_i x_j$. Then take the derivative of $f(\bf{x})$ with respect to a component $x_k$. We find \begin{eqnarray} \partial f/\partial x_k = f,_k = a_{ij} x_{i,k} x_j + a_{ij} x_i x_{j,k} = a_{ij} \delta_{ik} x_j + a_{ij} x_i \delta_{jk} = a_{kj} x_j + a_{ik} x_i, \end{eqnarray} which in matrix notation is $k$th component of ${\bf{x}}^T A + {\bf{x}}^T A^T$.

$\endgroup$
6
$\begingroup$

Yet another approach using the Frobenius product notation.

For a column vector $x \in \mathbb{R}^n$, and a matrix $A \in \mathbb{R}^{n \times n}$ we can write:

$$ x^TAx = Tr(x^TAx) = x:Ax$$

Then we take the differential and derivative as

\begin{align} d(x:Ax) & = dx:Ax + x:Adx\\ & = Ax:dx + A^Tx:dx\\ & = (Ax + A^Tx):dx\\ \frac{\partial (x^TAx)}{\partial x} &= (Ax + A^Tx) = (A + A^T)x \end{align}

$\endgroup$
3
$\begingroup$

I just learned a new trick when your independent variable is in more than two places within your formula: introduce a new (fake) parameter which will then disappear:

$$\frac{\partial}{\partial x} y^TAx = \frac{\partial y}{\partial x}[Ax]^T+y^TA $$ The transpose was to make the vector a row vector. Nothing deep there!

Now, if $y=x$ then $$ \frac{d}{dx} x^TAx = x^TA^T+x^TA = x^T(A+A^T) \ . $$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .