Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

How can I take derivative $$\frac{d}{dA}(x - Ab)(x - Ab)^T$$

where $x$ and $b$ are known vectors of the same size and matrix $A$ is symmetric and positive-definite?

Update: This expression could be expanded as $xx^T - Abx^T - xb^TA^T + Abb^TA^T$. Taking derivative will get $-2bx^T + \frac{d}{dA}Abb^TA^T$, so question now is how to calculate the last term.

share|improve this question
    
Did you try writing $i,j$-th element of your product and then differentiating it with respect to $A_{pq}$? –  TZakrevskiy Jun 16 '13 at 21:03
    
    
@TZakrevskiy, not yet, but it seems that I have to as I couldn't find any standard related formula –  sbos Jun 16 '13 at 21:09
    
@par, $x x^T$ is not scalar, it's a matrix, so I'm not sure that this would help me –  sbos Jun 16 '13 at 21:11
    
@sbos: Woops, misread. I guess it depends on how you define it then. I mean, I'm sure it's clear within the context of what you're doing. Maybe you want to take $\partial B_{m,n}/\partial A_{i,j}$ for all $m,n,i,j$ where $B=\left(x-Ab\right)\left(x-Ab\right)^T$? –  par Jun 16 '13 at 22:02
add comment

2 Answers

If you really want to take the derivative of the matrix $(x-Ab)(x-Ab)^T$ w.r.t. the matrix $A$, your result will be a matrix of matrices, because each derivative w.r.t. one matrix element $a_{ij}$ is a matrix. Your expansion

$$xx^T - Abx^T - xb^TA^T + Abb^TA^T$$

is correct, but your derivative isn't. You say that the derivative of $Abx^T$ is $bx^T$, but this cannot be true, since $Abx^T$ is a matrix and $bx^T$ is also a matrix, but it should be a matrix of matrices. Let's define

$$B=bx^T\quad\text{and}\quad C=bb^T$$

So you want the derivative of

$$-AB-B^TA^T+ACA$$

Let $a_{ij}$ and $b_{ij}$ denote the elements of matrices $A$ and $B$, respectively. Then we have

$$\frac{\partial(AB)_{ij}}{\partial a_{mn}}=\delta_{im}b_{nj}\\ \frac{\partial(B^TA^T)_{ij}}{\partial a_{mn}}=\delta_{jm}b_{ni}\\ \frac{\partial(ACA^T)_{ij}}{\partial a_{mn}}=\delta_{jm}(AC)_{in}+ \delta_{im}(AC^T)_{jn} $$

where $(.)_{ij}$ is the element with indices $i$ and $j$ of the matrix in parentheses, and $\delta_{ij}$ equals $1$ for $i=j$ and is zero otherwise. Note that in your case $C=C^T$.

share|improve this answer
2  
Thank you for your answer, Matt. Could you please explain the first step? How can you rewrite $(x - Ab)(x - Ab)^T$ as $(x - Ab)^T(x - Ab)$ and what is the reason for doing this? It seems that I miss something –  sbos Jun 16 '13 at 21:21
    
The derivative of a matrix with respect to another matrix is a tensor of fourth order, not of the second order. –  TZakrevskiy Jun 16 '13 at 21:31
    
I've updated my answer, see above. –  Matt L. Jun 17 '13 at 14:23
add comment

Let's use (http://en.wikipedia.org/wiki/Einstein_summation_convention).

$(Ab-x)_i = A_{ik}b_k-x_i$

$((x - Ab)^T(x - Ab))_{ij} = (A_{ik}b_k-x_i)(A_{js}b_s-x_j)$

$$\frac{\partial}{\partial A_{pq}}(A_{ik}b_k-x_i)(A_{js}b_s-x_j)=(A_{ik}b_k-x_i) \delta_{pj}\delta_{qs} b_s + \delta_{pi}\delta_{qk}b_k (A_{js}b_s-x_j)$$ $$ =(A_{ik}b_k-x_i) \delta_{pj} b_q + \delta_{pi} b_q (A_{js}b_s-x_j).$$

I don't know if there's a simplier way to express this derivative.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.