1
$\begingroup$

The mathematical expression is like this:

$f(\mu_{q(\beta)}, \Sigma_{q(\beta)}) = \mathbf{1}_{n}^{T} \exp \left \{ X \mu_{q(\beta)} + \frac{1}{2} \operatorname{diagonal}(X \Sigma_{q(\beta)} X^{T}) \right\}$ where $\exp$ indicates element-wise exponentiation and $\operatorname{diagonal}$ is a vector with the diagonal entries of the matrix as its components. Additionally, $\mathbf{1}_{n}^{T}$ is a vector in $\mathbb{R}^{n}$ with only ones as its entries and for notational ease, I used lower-case for vectors whereas matrices are in upper-case. $\Sigma_{q(\beta)}$ is symmetric whereas $X$ is not.

And I want to differentiate this by $\mu_{q(\beta)}$ and $\Sigma_{q(\beta)}$ each. So $\frac{\partial f}{\partial \mu_{q(\beta)}}$ and $\frac{\partial f}{\partial \Sigma_{q(\beta)}}$ are what I need.

$\endgroup$
1
$\begingroup$

Your elaborate subscripts are too hard to type, so I'm going to use simpler variables and write the problem as $$\eqalign{ w &= \mu_{q(\beta)} \cr S &= \Sigma_{q(\beta)} \cr y &= Xw+\frac{1}{2}\,{\rm diag}(XSX^T) \cr e &= \exp(y) \cr f &= 1^Te \cr }$$ The differentials of these quantities can be expressed using the Hadamard ($\circ$) product as $$\eqalign{ dy &= X\,dw+\frac{1}{2}\,{\rm diag}(X\,dS\,X^T) \cr de &= e\circ dy \cr df &= 1^Tde \cr &= 1^T(e\circ dy) \cr &= e^Tdy \cr &= e^T(X\,dw+\frac{1}{2}\,{\rm diag}(X\,dS\,X^T)) \cr }$$ Now set $dS=0$ and find the gradient with respect to $w$ $$\eqalign{ \frac{\partial f}{\partial w} &= e^TX \cr }$$ Working out the gradient with respect to $S$ is a bit harder $$\eqalign{ df &= \frac{1}{2}\,e^T{\rm diag}(X\,dS\,X^T)) \cr &= \frac{1}{2}\,{\rm diag}(e^T):X\,dS\,X^T \cr &= \frac{1}{2}\,E:X\,dS\,X^T \cr &= \frac{1}{2}\,X^TEX:dS \cr\cr \frac{\partial f}{\partial S} &= \frac{1}{2}\,X^TEX \cr &= \frac{1}{2}\,X^T\,{\rm diag}(\exp(y))\,X \cr }$$

$\endgroup$
  • $\begingroup$ Wow... This matrix calculus is extremely difficult. I could not have gotten it myself. Can I ask you how you studied Matrix Calculus and if there is any books or papers I can reference? $\endgroup$ – Daeyoung Lim Dec 2 '15 at 15:57
  • $\begingroup$ I found one problem here. The "diag" you used in the original equation that I wrote in the question was collecting the diagonal entries and making them into a vector. The last $\frac{\partial f}{\partial S}$ also has the "diag" operator only with the difference that it has a vector inside of it. Is that an operator that makes a diagonal matrix with the vector inside it? $\endgroup$ – Daeyoung Lim Dec 8 '15 at 0:49
  • $\begingroup$ I believe a matrix differentiation of a scalar function should be a matrix... $\endgroup$ – Daeyoung Lim Dec 8 '15 at 0:55
  • 1
    $\begingroup$ Yes, the name "diag" is ambiguous. When applied to a matrix argument, it returns the diagonal elements as a vector. When applied to a vector argument, it returns a diagonal matrix. Sometimes I write "Diag" (with a capital 'D') to distinguish the latter function. $\endgroup$ – lynn Dec 8 '15 at 5:22
  • 1
    $\begingroup$ Yes, your Hessian is correct. $\endgroup$ – lynn Dec 19 '15 at 23:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.