3
$\begingroup$

The mathematical expression is like this:

$f(\mu_{q(\beta)}, \Sigma_{q(\beta)}) = \mathbf{1}_{n}^{T} \exp \left \{ X \mu_{q(\beta)} + \frac{1}{2} \operatorname{diagonal}(X \Sigma_{q(\beta)} X^{T}) \right\}$ where $\exp$ indicates element-wise exponentiation and $\operatorname{diagonal}$ is a vector with the diagonal entries of the matrix as its components. Additionally, $\mathbf{1}_{n}^{T}$ is a vector in $\mathbb{R}^{n}$ with only ones as its entries and for notational ease, I used lower-case for vectors whereas matrices are in upper-case. $\Sigma_{q(\beta)}$ is symmetric whereas $X$ is not.

And I want to differentiate this by $\mu_{q(\beta)}$ and $\Sigma_{q(\beta)}$ each. So $\frac{\partial f}{\partial \mu_{q(\beta)}}$ and $\frac{\partial f}{\partial \Sigma_{q(\beta)}}$ are what I need.

$\endgroup$

1 Answer 1

3
$\begingroup$

Your elaborate subscripts are too hard to type, so I'm going to use simpler variables and write the problem as $$\eqalign{ w &= \mu_{q(\beta)} \cr S &= \Sigma_{q(\beta)} \cr y &= Xw+\frac{1}{2}\,{\rm diag}(XSX^T) \cr e &= \exp(y) \cr f &= 1^Te \cr }$$ The differentials of these quantities can be expressed using the Hadamard ($\circ$) product as $$\eqalign{ dy &= X\,dw+\frac{1}{2}\,{\rm diag}(X\,dS\,X^T) \cr de &= e\circ dy \cr df &= 1^Tde \cr &= 1^T(e\circ dy) \cr &= e^Tdy \cr &= e^T(X\,dw+\frac{1}{2}\,{\rm diag}(X\,dS\,X^T)) \cr }$$ Now set $dS=0$ and find the gradient with respect to $w$ $$\eqalign{ \frac{\partial f}{\partial w} &= e^TX \cr }$$ Working out the gradient with respect to $S$ is a bit harder $$\eqalign{ df &= \frac{1}{2}\,e^T{\rm diag}(X\,dS\,X^T)) \cr &= \frac{1}{2}\,{\rm diag}(e^T):X\,dS\,X^T \cr &= \frac{1}{2}\,E:X\,dS\,X^T \cr &= \frac{1}{2}\,X^TEX:dS \cr\cr \frac{\partial f}{\partial S} &= \frac{1}{2}\,X^TEX \cr &= \frac{1}{2}\,X^T\,{\rm diag}(\exp(y))\,X \cr }$$

$\endgroup$
9
  • $\begingroup$ Wow... This matrix calculus is extremely difficult. I could not have gotten it myself. Can I ask you how you studied Matrix Calculus and if there is any books or papers I can reference? $\endgroup$
    – Daeyoung
    Dec 2, 2015 at 15:57
  • $\begingroup$ I found one problem here. The "diag" you used in the original equation that I wrote in the question was collecting the diagonal entries and making them into a vector. The last $\frac{\partial f}{\partial S}$ also has the "diag" operator only with the difference that it has a vector inside of it. Is that an operator that makes a diagonal matrix with the vector inside it? $\endgroup$
    – Daeyoung
    Dec 8, 2015 at 0:49
  • $\begingroup$ I believe a matrix differentiation of a scalar function should be a matrix... $\endgroup$
    – Daeyoung
    Dec 8, 2015 at 0:55
  • 1
    $\begingroup$ Yes, the name "diag" is ambiguous. When applied to a matrix argument, it returns the diagonal elements as a vector. When applied to a vector argument, it returns a diagonal matrix. Sometimes I write "Diag" (with a capital 'D') to distinguish the latter function. $\endgroup$
    – lynn
    Dec 8, 2015 at 5:22
  • 1
    $\begingroup$ Yes, your Hessian is correct. $\endgroup$
    – lynn
    Dec 19, 2015 at 23:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .