1
$\begingroup$

Is there a way to calculate the derivative of a quadratic form $$ \frac{\partial x^TAx}{\partial x} = x^T(A + A^T) $$ using the chain rule of matrix differentiation? $$ \frac{\partial[UV]}{\partial x} = \frac{\partial U}{\partial x} V + U\frac{\partial V}{\partial x} $$ If not, what are the cases when the chain rule for matrix differentiation is applicable?

$\endgroup$
5
  • $\begingroup$ Are you derivating with respect to a vector? $\endgroup$
    – Botond
    Commented Jun 20, 2018 at 16:30
  • $\begingroup$ @Botond Yes, it's a vector. So basically math.stackexchange.com/questions/1007592/… is the best we can do? $\endgroup$
    – Konstantin
    Commented Jun 20, 2018 at 16:31
  • $\begingroup$ How do you define the derivation with respect to a vector? $\endgroup$
    – Botond
    Commented Jun 20, 2018 at 16:38
  • $\begingroup$ I think the "derivative with respect to a column matrix (here, vector)" is simply the row matrix of the derivatives with respect each of the vector entries. Is that right? $\endgroup$ Commented Jun 20, 2018 at 22:50
  • $\begingroup$ The rule you have written is not the chain rule, it is the product rule. $\endgroup$
    – littleO
    Commented Jun 21, 2018 at 0:24

2 Answers 2

6
$\begingroup$

Using the chain rule for matrix differentiation $$\frac{\partial[UV]}{\partial x} = \frac{\partial U}{\partial x}V + U\frac{\partial V}{\partial x}$$

But that is not the chain rule. That is the Leibniz (or product) rule. I assume that is what you meant. In that case the answer is yes. Here I show how to do it using index notation and Einstein summation convention.

Say our column vectors have upper indices, and our row vectors have lower indices, and our matrices have one upper and one lower index. In this notation, matrix multiplication $AB$ is a summation over a lower index of $A$ and an upper index of $B$: $${(AB)^{i}}_{j} = {A^{i}}_{k}{B^{k}}_{j} = {B^{k}}_{j}{A^{i}}_{k}$$ and the sum of matrices is given by $${(A+B)^{i}}_{j} = {A^{i}}_{j} + {B^{i}}_{j}$$ Then we use the kronecker delta $$\delta_{ij} =\begin{cases} 0 & i \neq j \\ 1 & i=j \end{cases}$$ to define the transpose $x^{T}$ of $x$ as $$(x^{T})_{i}=\delta_{ij}x^{j}$$ which implies $$x^{i} = \delta^{ij}(x^{T})_{j}$$ And we define the transpose $A^{T}$ of the matrix $A$ by $${(A^{T})^{i}}_{j} = \delta^{ik}{A^{l}}_{k}\delta_{jl}$$

Now you want $$\frac{\partial (x^{T}Ax)}{\partial x}\tag{1}$$ which I translate to index notation $$\left(\frac{\partial (x^{T}Ax)}{\partial x}\right)_{k} = \frac{\partial (x^{T}Ax)}{\partial x^{k}} = \frac{\partial [(x^{T})_{i}{A^{i}}_{j}x^{j}]}{\partial x^{k}}$$ and then use the product rule $$=\frac{\partial (x^{T})_{i}}{\partial x^{k}}{A^{i}}_{j}x^{j} +(x^{T})_{i}\frac{\partial {A^{i}}_{j}}{\partial x^{k}}x^{j} +(x^{T})_{i}{A^{i}}_{j}\frac{\partial x^{j}}{\partial x^{k}} \tag{2}$$

The partial derivative in the first summand on the right hand side is $$\frac{\partial (x^{T})_{i}}{\partial x^{k}} = \frac{\partial (x^{m}\delta_{mi})}{\partial x^{k}} = \frac{\partial x^{m}}{\partial x^{k}}\delta_{mi} = \delta^{m}_{k}\delta_{mi} = \delta_{ki}$$ where I used

  1. The definition of the transpose $(x^{T})_{i} = x^{m}\delta_{mi}$
  2. $\delta_{mi}$ is constant
  3. $\frac{\partial x^{m}}{\partial x^{k}} = \delta^{m}_{k}$

The second summand vanishes, because $\frac{\partial {A^{i}}_{j}}{\partial x^{k}} = 0$ since $A$ does not depend on $x$.

Using all this, equation $(2)$ reduces to

$$=\delta_{ki}{A^{i}}_{j}x^{j}+(x^{T})_{i}{A^{i}}_{j}\delta^{j}_{k}\\ =\delta_{ki}{A^{i}}_{j}\delta^{jl}(x^{T})_{l}+(x^{T})_{i}{A^{i}}_{k}\\ ={(A^{T})^{l}}_{k}(x^{T})_{l}+(x^{T})_{i}{A^{i}}_{k}\\ ={(A^{T})^{i}}_{k}(x^{T})_{i}+(x^{T})_{i}{A^{i}}_{k}\\ =(x^{T})_{i}\left({(A^{T})^{i}}_{k}+{A^{i}}_{k}\right)\\ =(x^{T})_{i}{\left(A^{T}+A\right)^{i}}_{k}\\ =\left(x^{T}(A^{T}+A)\right)_{k}\\ $$ Where I have used (starting from the second equal sign)

  1. $x^{i} = \delta^{ij}(x^{T})_{j}$
  2. The definition of the transpose ${(A^{T})^{l}}_{k} = \delta^{jl}{A^{i}}_{j}\delta_{ik}$
  3. Renaming the dummy index $l$ to $i$
  4. Factorize $(x^{T})_{i}$
  5. Sum of matrices
  6. Matrix multiplication (lower index of $x$ and upper index of $A^{T}+A$)

Hence

$$\left(\frac{\partial (x^{T}Ax)}{\partial x}\right)_{k} = \left(x^{T}(A^{T}+A)\right)_{k}$$

or, without the indices $$\frac{\partial (x^{T}Ax)}{\partial x} = x^{T}(A^{T}+A)$$


If you want to use purely matrix notation, without the indices, I guess you could say that $$\frac{\partial x^{T}}{\partial x}$$ is an operator that transposes the column vector immediately on its right. Then you could take equation $(1)$ and proceed as follows $$\frac{\partial (x^{T}Ax)}{\partial x} = \frac{\partial x^{T}}{\partial x}Ax + x^{T}\frac{\partial A}{\partial x}x + x^{T}A\frac{\partial x}{\partial x}\\ = (Ax)^{T} + x^{T}A\\ = x^{T}A^{T} + x^{T}A\\ = x^{T}(A^{T} + A)$$ But one (at least I) wouldn't have seen and understood this behaviour of $\frac{\partial x^{T}}{\partial x}$ were it not for the calculation in index notation.


I'll have to rectify: there is a way to see that behaviour of $\frac{\partial x^{T}v}{\partial x}$. If $v$ doesn't depend on $x$, we have $$\frac{\partial x^{T}v}{\partial x} = \frac{\partial x^{T}}{\partial x}v$$ But $x^{T}v = v^{T}x$, hence $$\frac{\partial x^{T}v}{\partial x} = \frac{\partial v^{T}x}{\partial x} = v^{T}\frac{\partial x}{\partial x} = v^{T}$$ Thus, we conclude $$\frac{\partial x^{T}}{\partial x}v = v^{T}$$

Similarly, we can check that $$v^{T}\frac{\partial x}{\partial x^{T}} = v$$

$\endgroup$
1
$\begingroup$

Background info: Recall that if $f:\mathbb R^n \to \mathbb R^m$ is differentiable at $x_0 \in \mathbb R^n$, then $f'(x_0)$ is an $m \times n$ matrix which satisfies $$ f(x) \approx f(x_0) + f'(x_0)(x - x_0). $$ The approximation is good when $x$ is close to $x_0$.


Derivation of product rule:

Suppose that $f:\mathbb R^n \to \mathbb R^m$ and $g:\mathbb R^n \to \mathbb R^m$ are differentiable at a point $x_0 \in \mathbb R^n$, and that $h:\mathbb R^n \to \mathbb R$ is defined by $$h(x) = \langle f(x), g(x) \rangle $$ for all $x \in \mathbb R^n$. Let's quickly derive a product rule to compute $h'(x_0)$. If $x$ is close to $x_0$, then \begin{align} h(x) &= \langle f(x), g(x) \rangle \\ &\approx \langle f(x_0) + f'(x_0)(x - x_0), g(x_0) + g'(x_0)(x - x_0) \rangle \\ &\approx \langle f(x_0), g(x_0) \rangle + \langle f'(x_0)(x - x_0), g(x_0) \rangle + \langle f(x_0), g'(x_0)(x - x_0) \rangle\\ &= h(x_0) + (g(x_0)^T f'(x_0) + f(x_0)^T g'(x_0))(x - x_0). \end{align} Comparing this with the equation $$ h(x) \approx h(x_0) + h'(x_0)(x - x_0), $$ we see that $$ \tag{1} \bbox[yellow,5px] { h'(x_0) = g(x_0)^T f'(x_0) + f(x_0)^T g'(x_0). } $$ This is our product rule.


Solution: Now let's consider the special case that $f(x) = x$ and $g(x) = Ax$, where $A$ is a real $n \times n$ matrix, so that $h(x) = \langle x, Ax \rangle$. The derivatives of $f$ and $g$ are given by $$ f'(x_0) = I, \qquad g'(x_0) = A. $$ (Here $I$ is the $n \times n$ identity matrix.) Using equation (1), we see that \begin{align} h'(x_0) &= (A x_0)^T I + x^T A \\ &= x_0^T(A^T + A). \end{align} This is what we wanted to show.

$\endgroup$
3
  • $\begingroup$ How do you go from $\langle f'(x_0)(x-x_0),g(x_0) \rangle$ to $g(x_0)^{T}f'(x_0)(x-x_0)$? I get $$\langle f'(x_0)(x-x_0),g(x_0) \rangle = [f'(x_0)(x-x_0)]^{T}g(x_0) = (x-x_0)^{T}f'(x_0)^{T}g(x_0) = [g(x_0)^{T}f'(x_0)(x-x_0)]^{T}$$ but I don't know how to continue from there. $\endgroup$ Commented Jun 21, 2018 at 2:41
  • $\begingroup$ @JackozeeHakkiuz In that case I used $\langle a,b\rangle = b^Ta$ rather than $a^T b$. $\endgroup$
    – littleO
    Commented Jun 21, 2018 at 3:42
  • $\begingroup$ Oh snap. Of course. Other way to see it is that $g(x_0)^{T}f'(x_0)(x-x_0)$ is just a scalar and hence it is equal to its transpose. But of course the commutativity of the inner product is much simpler. $\endgroup$ Commented Jun 21, 2018 at 4:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .