2
$\begingroup$

I am wondering how to calculate the gradient of a multivariate Normal respect to L where $\Sigma=LL^{T}$. I know what would be the derivative respect to $\Sigma$ and I can use chain rule to get the derivative respect to L but since $\Sigma$ is a matrix and L is matrix then derivative of a matrix respect to matrix is 4D tensor.

Therefore I am wondering can anyone help to find the derivative of

log|L|$-(x-z)L^{-1}L^{-T}(x-z)^{T}$ respect to L. The first term involving the determinant is a standard calculation but I am stuck in the second term.

$\endgroup$

1 Answer 1

2
$\begingroup$

In matrix calculus it is often easier to use differentials instead of the chain rule, since it avoids the need to calculate third and fourth order tensors.

Define the variables $$A = LL^T,\quad y = x-z$$ Note that $$A^T=A,\quad A^{-1} = L^{-T}L^{-1} $$ Write the cost function in terms of these new variables. $$\eqalign{ \phi &= \log(\det(L)) + y^TA^{-1}y \\ }$$ Then calculate the differential and the gradient. $$\eqalign{ d\phi &= d\log(\det(L)) + y^TdA^{-1}y \\ &= d{\,\rm tr}(\log(L)) + {\rm tr}\big(y^TdA^{-1}y\big) \\ &= {\rm tr}\Big(d\log(L) - y^TA^{-1}dA\,A^{-1}y\Big) \\ &={\rm tr}\Big(L^{-1}dL-y^TA^{-1}dL\,L^TA^{-1}y-y^TA^{-1}L\,dL^T\,A^{-1}y\Big) \\ &={\rm tr}\Big(L^{-1}dL-2L^TA^{-1}yy^TA^{-1}dL\Big) \\ \frac{\partial \phi}{\partial L} &= L^{-1} - 2L^TA^{-1}yy^TA^{-1} \\ &= L^{-1} - 2L^{-1}yy^TL^{-T}L^{-1} \\ }$$ Depending on your preferred layout convention, you may want the transpose of this result.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .