I am trying to understand the proof for the Kullback-Leibler divergence between two multivariate normal distributions. On the way, a sort of trace trick is applied for the expectation of the quadratic form $$E[ (x-\mu)^T \Sigma^{-1} (x-\mu) ]= \operatorname{trace}(E[(x-\mu)(x-\mu)^T)] \Sigma^{-1}),$$

where $x$ is MV-normal with mean $\mu$ and covariance matrix $\Sigma$. The expectation is taken over $x$.

I would like to understand why this identity holds. I think more than one step is taken at once. I believe, $\operatorname{trace}(E[(x-\mu)(x-\mu)^T] \Sigma^{-1})$ = $\operatorname{trace}(E[(x-\mu) \Sigma^{-1} (x-\mu)^T])$, but where does the trace come from?

  • 5
    $\begingroup$ The "trace trick" is the cyclic property of the trace operator: If the matrix products make sense, then $\operatorname{tr} (ABC) = \operatorname{tr}(CAB) = \operatorname{tr}(BCA)$. $\endgroup$
    – grand_chat
    Commented Apr 10, 2017 at 23:30
  • $\begingroup$ Remember that a quadratic form is always a scalar (1x1 matrix). $\endgroup$
    – igorkf
    Commented Mar 10 at 2:09

2 Answers 2


Where does the trace come from?

A real number can be thought of as a $1 \times 1$ matrix, and its trace is itself. Thus $$(x-\mu)^\top \Sigma^{-1} (x-\mu) = \operatorname{tr}\left((x-\mu)^\top \Sigma^{-1} (x-\mu)\right)$$

More than one step is taken at once.

After applying the above step, use the cyclic property of the trace to obtain $$\operatorname{tr}\left((x-\mu)^\top \Sigma^{-1} (x-\mu)\right) = \operatorname{tr}\left((x-\mu)(x-\mu)^\top \Sigma^{-1} \right)$$ By linearity of the trace operator, you can push the expectation inside $$E \operatorname{tr}\left((x-\mu)(x-\mu)^\top \Sigma^{-1} \right) = \operatorname{tr}\left(E\left[(x-\mu)(x-\mu)^\top\right] \Sigma^{-1} \right).$$


To add to @angryavian's answer, you can swap expectation and trace because, $$ tr(E[A]) = tr(\begin{bmatrix} E[a_{11}] & \dots & \\ \vdots & E[a_{22}] & \\ & & \ddots\end{bmatrix}) = \sum\limits_{i=1}^N E[a_{ii}]=E[\sum\limits_{i=1}^N a_{ii}] = E[tr(A)] $$


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .