This question is an extension of this question. The arbitrary function $B(\cdot)$ is now as specified as follows: $$ B(\mu, \sigma^{2}) = \exp \left(\mu + \frac{1}{2}\sigma^{2} \right). $$ So if I write $$ \begin{align*} s &= X_{i}\beta + Z_{i}\mu_{i} + \frac{1}{2}\operatorname{dg}\left(Z_{i}\Lambda_{i}Z_{i}^{T} \right)\\ e &= \exp \left(s \right)\\ E &= \operatorname{Diag}\left(e\right) \end{align*} $$ where $\exp$ is applied element wise and $\operatorname{Diag}$ is an operator that creates a diagonal matrix with the elements of the vector inside.

The problem is that I need to compute $\frac{\partial^{2} \underline{\ell}}{\partial \operatorname{vech}\left( \Lambda_{i}\right)\partial\mu_{i} }$, I'm stuck with calculating the differential within the $\operatorname{Diag}$ operator.

$$ \frac{\partial \underline{\ell} }{\partial \operatorname{vech}\left(\Lambda_{i} \right)} = \frac{1}{2}\left(\operatorname{vec} \left(\Lambda_{i}^{-1} - \Sigma^{-1} - Z_{i}^{T}EZ_{i} \right)\right)^{T}D_{K}. $$ Setting $f= \frac{\partial \underline{\ell} }{\partial \operatorname{vech}\left(\Lambda_{i} \right)}$ and trying to differentiate this again with respect to $\mu_{i}$, $$ \begin{align*} df &= -\frac{1}{2} \left(\operatorname{vec} \left(Z_{i}^{T} dE Z_{i} \right)\right)^{T}D_{K}\\ dE &= \operatorname{Diag}\left(de \right)\\ &= \operatorname{Diag}\left(e \circ ds \right)\\ &= \operatorname{Diag} \left(e \circ Z_{i}d\mu_{i} \right) \end{align*} $$ the differential is inside the $\operatorname{Diag}$ operator. How do I proceed from here?

  • $\begingroup$ Does the basis you have in mind depend on your variables in any way? If not, then the "Diag" operator commutes with derivative operators $\endgroup$
    – co9olguy
    Jan 8, 2016 at 11:07
  • $\begingroup$ @co9olguy yeah, but in order for me to complete the Hessian, $d\mu_{i}$ has to be factored out alone... $\endgroup$
    – Daeyoung
    Jan 8, 2016 at 11:11
  • $\begingroup$ If the basis for "Diag" is a function of your variables $f(\mu_i)$, then I suppose you'll have to use the chain rule to access the inside arguments. This would likely require you to know the functional dependence of the basis on the variables $\endgroup$
    – co9olguy
    Jan 8, 2016 at 11:16
  • $\begingroup$ @co9olguy The basis for "Diag" is e which is described in my question and I need to compute $df$ with respect to $d\mu_{i}$. The chain rule has been applied once for the exponential term and was expressed with Hadamard product: $e \circ ds$. $\endgroup$
    – Daeyoung
    Jan 8, 2016 at 11:32

1 Answer 1


Ok, I did some research and got the answer for the question so I will post an answer for my own question. So we start with $df = -\frac{1}{2}\left(\operatorname{vec}\left(Z_{i}^{T}dEZ_{i} \right) \right)^{T}D_{K}$ and $dE = \operatorname{Diag}\left(de\right)$. We will use the following theorem: $$ \operatorname{vec}\left(A^{T}\operatorname{Diag}\left(b\right)A \right) = \mathcal{Q}\left(A\right)^{T}b $$ where $\mathcal{Q}\left(A\right) = \left(A \otimes \boldsymbol{1}^{T}\right) \circ \left(\boldsymbol{1}^{T}\otimes A\right)$. $\boldsymbol{1}$ is a $d \times 1$ vector of ones when $A$ is $n \times d$ matrix. ($\otimes$ is the Kronecker product and $\circ$ is the Hadamard product.) Then, $$ \begin{align*}df &= -\frac{1}{2}\left(\operatorname{vec}\left(Z_{i}^{T}\operatorname{Diag}\left(de\right)Z_{i} \right) \right)^{T}D_{K} \\ &= -\frac{1}{2}\left(de\right)^{T}\, \mathcal{Q}\left(Z_{i}\right)D_{K}\\ &= -\frac{1}{2}\left(e\circ ds\right)^{T} \, \mathcal{Q}\left(Z_{i}\right)D_{K} \\ &= -\frac{1}{2}\left(\operatorname{Diag}\left(e\right) Z_{i} \, d\mu_{i} \right)^{T} \, \mathcal{Q}\left(Z_{i}\right)D_{K} \\ &= -\frac{1}{2}\left(d\mu_{i}\right)^{T}Z_{i}^{T}E\, \mathcal{Q}\left(Z_{i}\right)D_{K} \end{align*} $$ Thus, $$ \frac{\partial^{2} \ell}{\partial \operatorname{vech}\left(\Lambda_{i}\right) \partial \mu_{i}}=-\frac{1}{2}Z_{i}^{T}E\, \mathcal{Q}\left(Z_{i}\right)D_{K}. $$

For those of you who want the proof of the theorem I used, please refer to M.P. Wand 2013, "Fully Simplified Multivariate Normal Updates in Non-Conjugate Variational Message Passing".

  • 3
    $\begingroup$ Wand's theorem is interesting. Some experimentation yielded a generalization which might prove useful $${\rm vec}(A\,{\rm Diag}(b)\,C) = (C^T\otimes 1_a)\circ(1_c\otimes A)\,b$$where the dimensions of the ones vectors are such that the products $(A^T\,1_a)$ and $(C\,1_c)$ are defined. $\endgroup$
    – lynn
    Mar 22, 2016 at 2:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .