8
$\begingroup$

I am trying to understand the proof for the Kullback-Leibler divergence between two multivariate normal distributions. On the way, a sort of trace trick is applied for the expectation of the quadratic form $$E(( (x-\mu)^T \Sigma^{-1} (x-\mu) )= \text{tr}(E((x-\mu)(x-\mu)^T) \Sigma^{-1},$$

where $x$ is MV-normal with mean $\mu$ and covariance matrix $\Sigma$. The expectation is taken over $x$.

I would like to understand why this identity holds. I think more than one step is taken at once. I believe, $\text{tr}(E((x-\mu)(x-\mu)^T) \Sigma^{-1}$ = $\text{tr}(E((x-\mu) \Sigma^{-1} (x-\mu)^T)$, but where does the trace come from?

$\endgroup$
  • 2
    $\begingroup$ The "trace trick" is the cyclic property of the trace operator: If the matrix products make sense, then $\operatorname{tr} (ABC) = \operatorname{tr}(CAB) = \operatorname{tr}(BCA)$. $\endgroup$ – grand_chat Apr 10 '17 at 23:30
9
$\begingroup$

Where does the trace come from?

A real number can be thought of as a $1 \times 1$ matrix, and its trace is itself. Thus $$(x-\mu)^\top \Sigma^{-1} (x-\mu) = \operatorname{tr}\left((x-\mu)^\top \Sigma^{-1} (x-\mu)\right)$$

More than one step is taken at once.

After applying the above step, use the trace trick to obtain $$\operatorname{tr}\left((x-\mu)^\top \Sigma^{-1} (x-\mu)\right) = \operatorname{tr}\left((x-\mu)(x-\mu)^\top \Sigma^{-1} \right)$$ By linearity of the trace operator, you can push the expectation inside $$E \operatorname{tr}\left((x-\mu)(x-\mu)^\top \Sigma^{-1} \right) = \operatorname{tr}\left(E\left[(x-\mu)(x-\mu)^\top\right] \Sigma^{-1} \right).$$

$\endgroup$
1
$\begingroup$

To add to @angryavian's answer, you can swap expectation and trace because, $$ tr(E[A]) = tr(\begin{bmatrix} E[a_{11}] & \dots & \\ \vdots & E[a_{22}] & \\ & & \ddots\end{bmatrix}) = \sum\limits_{i=1}^N E[a_{ii}]=E[\sum\limits_{i=1}^N a_{ii}] = E[tr(A)] $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.