Derivative of column-row multiplication How can I take derivative $$\frac{d}{dA}(x - Ab)(x - Ab)^T$$
where $x$ and $b$ are known vectors of the same size and matrix $A$ is symmetric and positive-definite?
Update:
This expression could be expanded as $xx^T - Abx^T - xb^TA^T + Abb^TA^T$. Taking derivative will get $-2bx^T + \frac{d}{dA}Abb^TA^T$, so question now is how to calculate the last term.
 A: If you really want to take the derivative of the matrix $(x-Ab)(x-Ab)^T$ w.r.t. the matrix $A$, your result will be a matrix of matrices, because each derivative w.r.t. one matrix element $a_{ij}$ is a matrix. Your expansion
$$xx^T - Abx^T - xb^TA^T + Abb^TA^T$$
is correct, but your derivative isn't. You say that the derivative of $Abx^T$ is $bx^T$, but this cannot be true, since $Abx^T$ is a matrix and $bx^T$ is also a matrix, but it should be a matrix of matrices. Let's define
$$B=bx^T\quad\text{and}\quad C=bb^T$$
So you want the derivative of
$$-AB-B^TA^T+ACA$$
Let $a_{ij}$ and $b_{ij}$ denote the elements of matrices $A$ and $B$, respectively. Then we have
$$\frac{\partial(AB)_{ij}}{\partial a_{mn}}=\delta_{im}b_{nj}\\
\frac{\partial(B^TA^T)_{ij}}{\partial a_{mn}}=\delta_{jm}b_{ni}\\
\frac{\partial(ACA^T)_{ij}}{\partial a_{mn}}=\delta_{jm}(AC)_{in}+
\delta_{im}(AC^T)_{jn}
$$
where $(.)_{ij}$ is the element with indices $i$ and $j$ of the matrix in parentheses, and $\delta_{ij}$ equals $1$ for $i=j$ and is zero otherwise. Note that in your case $C=C^T$.
A: Let's use (http://en.wikipedia.org/wiki/Einstein_summation_convention).
$(Ab-x)_i = A_{ik}b_k-x_i$
$((x - Ab)^T(x - Ab))_{ij} = (A_{ik}b_k-x_i)(A_{js}b_s-x_j)$
$$\frac{\partial}{\partial A_{pq}}(A_{ik}b_k-x_i)(A_{js}b_s-x_j)=(A_{ik}b_k-x_i) \delta_{pj}\delta_{qs} b_s  +  \delta_{pi}\delta_{qk}b_k (A_{js}b_s-x_j)$$
$$ =(A_{ik}b_k-x_i) \delta_{pj}  b_q  +  \delta_{pi} b_q (A_{js}b_s-x_j).$$
I don't know if there's a simplier way to express this derivative.
