# Derivative of column-row multiplication

How can I take derivative $$\frac{d}{dA}(x - Ab)(x - Ab)^T$$

where $x$ and $b$ are known vectors of the same size and matrix $A$ is symmetric and positive-definite?

Update: This expression could be expanded as $xx^T - Abx^T - xb^TA^T + Abb^TA^T$. Taking derivative will get $-2bx^T + \frac{d}{dA}Abb^TA^T$, so question now is how to calculate the last term.

-
Did you try writing $i,j$-th element of your product and then differentiating it with respect to $A_{pq}$? – TZakrevskiy Jun 16 '13 at 21:03
@TZakrevskiy, not yet, but it seems that I have to as I couldn't find any standard related formula – sbos Jun 16 '13 at 21:09
@par, $x x^T$ is not scalar, it's a matrix, so I'm not sure that this would help me – sbos Jun 16 '13 at 21:11
@sbos: Woops, misread. I guess it depends on how you define it then. I mean, I'm sure it's clear within the context of what you're doing. Maybe you want to take $\partial B_{m,n}/\partial A_{i,j}$ for all $m,n,i,j$ where $B=\left(x-Ab\right)\left(x-Ab\right)^T$? – par Jun 16 '13 at 22:02

If you really want to take the derivative of the matrix $(x-Ab)(x-Ab)^T$ w.r.t. the matrix $A$, your result will be a matrix of matrices, because each derivative w.r.t. one matrix element $a_{ij}$ is a matrix. Your expansion

$$xx^T - Abx^T - xb^TA^T + Abb^TA^T$$

is correct, but your derivative isn't. You say that the derivative of $Abx^T$ is $bx^T$, but this cannot be true, since $Abx^T$ is a matrix and $bx^T$ is also a matrix, but it should be a matrix of matrices. Let's define

$$B=bx^T\quad\text{and}\quad C=bb^T$$

So you want the derivative of

$$-AB-B^TA^T+ACA$$

Let $a_{ij}$ and $b_{ij}$ denote the elements of matrices $A$ and $B$, respectively. Then we have

$$\frac{\partial(AB)_{ij}}{\partial a_{mn}}=\delta_{im}b_{nj}\\ \frac{\partial(B^TA^T)_{ij}}{\partial a_{mn}}=\delta_{jm}b_{ni}\\ \frac{\partial(ACA^T)_{ij}}{\partial a_{mn}}=\delta_{jm}(AC)_{in}+ \delta_{im}(AC^T)_{jn}$$

where $(.)_{ij}$ is the element with indices $i$ and $j$ of the matrix in parentheses, and $\delta_{ij}$ equals $1$ for $i=j$ and is zero otherwise. Note that in your case $C=C^T$.

-
Thank you for your answer, Matt. Could you please explain the first step? How can you rewrite $(x - Ab)(x - Ab)^T$ as $(x - Ab)^T(x - Ab)$ and what is the reason for doing this? It seems that I miss something – sbos Jun 16 '13 at 21:21
The derivative of a matrix with respect to another matrix is a tensor of fourth order, not of the second order. – TZakrevskiy Jun 16 '13 at 21:31
I've updated my answer, see above. – Matt L. Jun 17 '13 at 14:23

$(Ab-x)_i = A_{ik}b_k-x_i$

$((x - Ab)^T(x - Ab))_{ij} = (A_{ik}b_k-x_i)(A_{js}b_s-x_j)$

$$\frac{\partial}{\partial A_{pq}}(A_{ik}b_k-x_i)(A_{js}b_s-x_j)=(A_{ik}b_k-x_i) \delta_{pj}\delta_{qs} b_s + \delta_{pi}\delta_{qk}b_k (A_{js}b_s-x_j)$$ $$=(A_{ik}b_k-x_i) \delta_{pj} b_q + \delta_{pi} b_q (A_{js}b_s-x_j).$$

I don't know if there's a simplier way to express this derivative.

-