Matrix derivative of structured matrix (with constraint) Let
$$c (A,B) := \log | ABA' + I - \mbox{diag}(ABA')|$$
where matrix $A$ is not necessarily square, matrix $B$ is symmetric, and $I$ is the identity matrix. How to obtain the derivatives $\frac{dc}{dA}$ and $\frac{dc}{dB}$?
 A: Use $(\odot)$ to represent the elementwise/Hadamard product and $(:)$ for the trace/Frobenius product, i.e.
$$A:B = {\rm Tr}(A^TB)$$
Define the matrices
$$\eqalign{
I &= {\rm identity\,matrix} \cr
J &= {\rm matrix\,of\,all\,ones} \cr
F &= J-I \cr
X &= ABA^T \cr
Y &= I + F\odot X \cr
Z &= F\odot Y^{-1} =\,\, Y^{-1} - {\rm diag}(Y^{-1}) \cr
}$$
NB: $\,$ All of these matrices are symmetric.
Write the function in terms of these new variables. Then find its differential and gradients.
$$\eqalign{
c &= \log(\det(Y)) \cr
dc
 &= Y^{-1}:dY \cr
 &= Y^{-1}:(F\odot dX) \cr
 &= Z:dX \cr
 &= Z:(dA\,B\,A^T + A\,B\,dA^T + A\,dB\,A^T) \cr
 &= (ZAB^T+Z^TAB):dA + (A^TZA):dB \cr
 &= 2ZAB:dA \,\,+\,\, A^TZA:dB \cr
\frac{\partial c}{\partial A} &= 2ZAB,\quad
\frac{\partial c}{\partial B} = A^TZA \cr\cr
}$$
To follow the above derivation, you need to know a couple of things.
$1)$ The cyclic property of the trace allows the terms in a Frobenius product to be rearranged in a number of different ways, e.g.
$$\eqalign{
A:BC = B^TA:C = AC^T:B = etc
}$$
$2)$ The Hadamard and Frobenius products are commutative and mutually commutative
$$\eqalign{
A:B &= B:A \cr
A\odot B &= B\odot A \cr
A:(B\odot C) &= (B\odot A):C =\,\, (A\odot B\odot C): J \cr
}$$
