This question is an extension of this question. The arbitrary function $B(\cdot)$ is now as specified as follows: $$ B(\mu, \sigma^{2}) = \exp \left(\mu + \frac{1}{2}\sigma^{2} \right). $$ So if I write $$ \begin{align*} s &= X_{i}\beta + Z_{i}\mu_{i} + \frac{1}{2}\operatorname{dg}\left(Z_{i}\Lambda_{i}Z_{i}^{T} \right)\\ e &= \exp \left(s \right)\\ E &= \operatorname{Diag}\left(e\right) \end{align*} $$ where $\exp$ is applied element wise and $\operatorname{Diag}$ is an operator that creates a diagonal matrix with the elements of the vector inside.

The problem is that I need to compute $\frac{\partial^{2} \underline{\ell}}{\partial \operatorname{vech}\left( \Lambda_{i}\right)\partial\mu_{i} }$, I'm stuck with calculating the differential within the $\operatorname{Diag}$ operator.

$$ \frac{\partial \underline{\ell} }{\partial \operatorname{vech}\left(\Lambda_{i} \right)} = \frac{1}{2}\left(\operatorname{vec} \left(\Lambda_{i}^{-1} - \Sigma^{-1} - Z_{i}^{T}EZ_{i} \right)\right)^{T}D_{K}. $$ Setting $f= \frac{\partial \underline{\ell} }{\partial \operatorname{vech}\left(\Lambda_{i} \right)}$ and trying to differentiate this again with respect to $\mu_{i}$, $$ \begin{align*} df &= -\frac{1}{2} \left(\operatorname{vec} \left(Z_{i}^{T} dE Z_{i} \right)\right)^{T}D_{K}\\ dE &= \operatorname{Diag}\left(de \right)\\ &= \operatorname{Diag}\left(e \circ ds \right)\\ &= \operatorname{Diag} \left(e \circ Z_{i}d\mu_{i} \right) \end{align*} $$ the differential is inside the $\operatorname{Diag}$ operator. How do I proceed from here?

  • $\begingroup$ Does the basis you have in mind depend on your variables in any way? If not, then the "Diag" operator commutes with derivative operators $\endgroup$ – co9olguy Jan 8 '16 at 11:07
  • $\begingroup$ @co9olguy yeah, but in order for me to complete the Hessian, $d\mu_{i}$ has to be factored out alone... $\endgroup$ – Daeyoung Lim Jan 8 '16 at 11:11
  • $\begingroup$ If the basis for "Diag" is a function of your variables $f(\mu_i)$, then I suppose you'll have to use the chain rule to access the inside arguments. This would likely require you to know the functional dependence of the basis on the variables $\endgroup$ – co9olguy Jan 8 '16 at 11:16
  • $\begingroup$ @co9olguy The basis for "Diag" is e which is described in my question and I need to compute $df$ with respect to $d\mu_{i}$. The chain rule has been applied once for the exponential term and was expressed with Hadamard product: $e \circ ds$. $\endgroup$ – Daeyoung Lim Jan 8 '16 at 11:32

Ok, I did some research and got the answer for the question so I will post an answer for my own question. So we start with $df = -\frac{1}{2}\left(\operatorname{vec}\left(Z_{i}^{T}dEZ_{i} \right) \right)^{T}D_{K}$ and $dE = \operatorname{Diag}\left(de\right)$. We will use the following theorem: $$ \operatorname{vec}\left(A^{T}\operatorname{Diag}\left(b\right)A \right) = \mathcal{Q}\left(A\right)^{T}b $$ where $\mathcal{Q}\left(A\right) = \left(A \otimes \boldsymbol{1}^{T}\right) \circ \left(\boldsymbol{1}^{T}\otimes A\right)$. $\boldsymbol{1}$ is a $d \times 1$ vector of ones when $A$ is $n \times d$ matrix. ($\otimes$ is the Kronecker product and $\circ$ is the Hadamard product.) Then, $$ \begin{align*}df &= -\frac{1}{2}\left(\operatorname{vec}\left(Z_{i}^{T}\operatorname{Diag}\left(de\right)Z_{i} \right) \right)^{T}D_{K} \\ &= -\frac{1}{2}\left(de\right)^{T}\, \mathcal{Q}\left(Z_{i}\right)D_{K}\\ &= -\frac{1}{2}\left(e\circ ds\right)^{T} \, \mathcal{Q}\left(Z_{i}\right)D_{K} \\ &= -\frac{1}{2}\left(\operatorname{Diag}\left(e\right) Z_{i} \, d\mu_{i} \right)^{T} \, \mathcal{Q}\left(Z_{i}\right)D_{K} \\ &= -\frac{1}{2}\left(d\mu_{i}\right)^{T}Z_{i}^{T}E\, \mathcal{Q}\left(Z_{i}\right)D_{K} \end{align*} $$ Thus, $$ \frac{\partial^{2} \ell}{\partial \operatorname{vech}\left(\Lambda_{i}\right) \partial \mu_{i}}=-\frac{1}{2}Z_{i}^{T}E\, \mathcal{Q}\left(Z_{i}\right)D_{K}. $$

For those of you who want the proof of the theorem I used, please refer to M.P. Wand 2013, "Fully Simplified Multivariate Normal Updates in Non-Conjugate Variational Message Passing".

  • 2
    $\begingroup$ Wand's theorem is interesting. Some experimentation yielded a generalization which might prove useful $${\rm vec}(A\,{\rm Diag}(b)\,C) = (C^T\otimes 1_a)\circ(1_c\otimes A)\,b$$where the dimensions of the ones vectors are such that the products $(A^T\,1_a)$ and $(C\,1_c)$ are defined. $\endgroup$ – lynn Mar 22 '16 at 2:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.