5
$\begingroup$

What is the derivative of the symmetric bilinear form $$ f_X(\mathbf{a},\mathbf{b}) = \mathbf{a}^T X \mathbf{b} $$ with respect to the (symmetric) matrix $X$?

Following Wikipedia, and using the denominator layout, I would say $$ \frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b}) = \mathbf{a}\mathbf{b}^T $$

But since $f_X$ is symmetric, $f_X(\mathbf{a},\mathbf{b}) = f_X(\mathbf{b},\mathbf{a})$. If I derive this equality I obtain $$ \frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b}) = \frac{\partial f_X}{\partial X}(\mathbf{b},\mathbf{a}) \qquad\Rightarrow\qquad \mathbf{a}\mathbf{b}^T = \mathbf{b}\mathbf{a}^T $$ which is wrong because $\mathbf{a}\mathbf{b}^T \neq \mathbf{b}\mathbf{a}^T$

Where am I wrong?

$\endgroup$
  • $\begingroup$ The function defined by $f_{a,b}(X):=\langle a, Xb \rangle$ is clearly linear, therefore $\partial f_{a,b}(X)H=f_{a,b}(H)$, but if you want the derivative of $f_X(a,b):=\langle a, Xb\rangle$ then you have that $\partial f_X(a,b)(c,d)=f_X(a,d)+f_X(c,b)$ $\endgroup$ – Masacroso Apr 7 at 9:09
  • $\begingroup$ You might find this post interesting. $\endgroup$ – greg Apr 8 at 4:40
2
$\begingroup$

Wikipedia has calculated the gradient with respect to an unconstrained $X$ matrix, i.e. $$G = ab^T$$ To enforce the symmetry constraint, the result should be modified to $$\eqalign{ G_{sym} &= G+G^T-I\odot G \\&= 2\operatorname{Sym}(G)-\operatorname{Diag}(G) }$$ This is in the Matrix Cookbook in section 2.8

So, for the current problem $$G_{sym} = ab^T + ba^T - \operatorname{Diag}(a)\cdot\operatorname{Diag}(b)$$

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

The problem is that the so-called 'denominator layout'

$$\frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b}) = \mathbf{a}\mathbf{b}^T$$

is just a very special shorthand notation for the general directional derivative

$$\frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b})[H] = \mathbf{a}^TH\mathbf{b}$$

where $H$ is the direction into which you are deriving, i.e. a general operator between the source and destination spaces. In particular, $H$ is not restricted to being a symmetric operator!

You can see the connection between directional derivative and 'denominator layout' symbolism by writing the bilinear form as a trace. The trace can be permuted cyclically (first step) and is invariant under transposition of its argument (second step):

$$\mathbf{a}^TH\mathbf{b}=\rm{tr}(H\mathbf{b}\mathbf{a}^T)=\rm{tr}(\mathbf{a}\mathbf{b}^TH^T)$$

Hence, the 'denominator layout' matrix derivative is the kernel $\mathbf{a}\mathbf{b}^T$ of the trace contained in the directional derivative with respect to $H^T$. No surprise, you were getting a little confused...

If you are sticking closely to the directional derivative, everything is clearer. Since your very special direction $H$ is not an arbitrary operator, but a symmetric one, according to your presumptions, we have

$$H=\frac{1}{2}(H+H^T)$$

and hence,

$$\frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b})[H] = \frac{1}{2}(\mathbf{a}^T(H+H^T)\mathbf{b})=\frac{1}{2}\mathbf{a}^TH\mathbf{b}+\frac{1}{2}\mathbf{b}^TH\mathbf{a}$$

where in the last step the invariance under transposition has been used again. So, finally, we obtain the directional derivative of your bilinear form for the special case of a symmetric $X$ and $H$ respectively:

$$\frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b})[H] = \rm{tr}\left(\frac{1}{2}\left(\mathbf{b}\mathbf{a}^T+\mathbf{a}\mathbf{b}^T\right)H\right)$$

More specifically, the question 'Where am I wrong?' can now be answered:

You used the symmetry constraint $X^T=X$ when you computed the derivative, but you did not apply the symmetry constraint to the (hidden, in your preferred notation) directional argument $H$ as well. Instead, by not paying attention to $H$, you have treated it as if it was still a completely general operator. You were vulnerable to this issue because the 'denominator layout' notation omits the direction $H$ into which you actually derive.

By applying the constraint to the directional argument as well, you would have obtained the derivative in the symmetrized form in the first place, as you can see above, and not in the unsymmetric form that leads to the false conclusion

$$\mathbf{b}\mathbf{a}^T=\mathbf{a}\mathbf{b}^T.$$

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

The partial (i.e. unconstrained) derivative $$\frac{\partial f(X)}{\partial X}=Y$$ is given by

$$Y_{i,j}= \frac{\partial f(X)}{\partial X_{i,j}}$$

For $f(X)=a^T X b =\sum_{i,j} a_i X_{i,j} b_j$ this gives $Y_{i,j}=a_i b_j$ or $Y= a b^T$.

But here we are interested in the total derivative. Because the matrix is symmetric we have the restriction $dX_{i,j} = dX_{j,i}$ and

$$d f(X)= \begin{cases} \frac{\partial f(X)}{\partial X_{i,j}} d X_{i,j} + \frac{\partial f(X)}{\partial X_{j,i}} d X_{j,i} & i\ne j\\ \frac{\partial f(X)}{\partial X_{i,j}} d X_{i,j} & i=j \end{cases} $$

Hence in our case

$$ \frac{d f(X)}{dX_{i,j}}= \begin{cases} \frac{\partial f(X)}{\partial X_{i,j}} + \frac{\partial f(X)}{\partial X_{j_,i}} = a_i b_j + b_i a_j & i\ne j\\ \frac{\partial f(X)}{\partial X_{i,j}} = a_i b_i & i=j \end{cases} $$

This can be written as

$$\frac{d f(X)}{dX}= a^T b + b^T a - diag(a \circ b)$$

See also here

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.