2
$\begingroup$

I am considering a variance of two forms:

  1. $ R(x) = (x-m)^\top A (x-m) + b^\top (x-m) + c $
  2. $ R'(\Delta) = \Delta^\top A \Delta + b^\top \Delta + c $

where $x$ is a random variable of $\mathcal{N}(m,\Sigma)$ (Gaussian of mean $m$ and variance $\Sigma$). $\Delta = x-m$, so it is a random variable of $\mathcal{N}(0,\Sigma)$. $A$ is symmetric ($A^\top=A$).

Now, the two equations are the same, so $\mathrm{var}[R]$ and $\mathrm{var}[R']$ should be the same. But when I tried to derive them, the results were different. Please help me to find out this mystery.

Preliminary.

$$\begin{align} \mathrm{var}[x^\top A x] &= 2\mathrm{Tr}(A\Sigma A\Sigma) + 4 m^\top A \Sigma A m \\ \mathrm{var}[b^\top x] &= b^\top \Sigma b \end{align}$$

Case 2.

$$\begin{align} \mathrm{var}[R'] & = \mathrm{var}[\Delta^\top A \Delta] + \mathrm{var}[b^\top \Delta] \\ & = 2\mathrm{Tr}(A\Sigma A\Sigma) + b^\top \Sigma b \end{align}$$

Case 1.

$$\begin{align} R(x) &= (x-m)^\top A (x-m) + b^\top (x-m) + c \\ &= x^\top A x + (b^\top - 2m^\top A) x + \mathit{constant} \end{align}$$

$$\begin{align} \mathrm{var}[R] &= \mathrm{var}[x^\top A x] + \mathrm{var}[(b^\top - 2m^\top A) x] \\ &= 2\mathrm{Tr}(A\Sigma A\Sigma) + 4 m^\top A \Sigma A m + (b^\top - 2m^\top A) \Sigma (b - 2A m) \\ &= 2\mathrm{Tr}(A\Sigma A\Sigma) + 4 m^\top A \Sigma A m \\ &\quad + b^\top \Sigma b - 2m^\top A \Sigma b - 2 b^\top \Sigma A m + 4 m^\top A \Sigma A m \end{align}$$

Therefore, unless $4 m^\top A \Sigma A m - 2m^\top A \Sigma b - 2 b^\top \Sigma A m + 4 m^\top A \Sigma A m$ is zero, the case 1 and 2 are not the same.

Where is the problem? Thank you in advance.

$\endgroup$
  • $\begingroup$ I have understood the problem: var[x+y] != var[x] + var[y] when x and y are not independent (the above case). I will write the correct answer later. $\endgroup$ – Akihiko May 29 '15 at 4:43
5
$\begingroup$

The problem is that $\mathrm{var}[x+y] \neq \mathrm{var}[x] + \mathrm{var}[y]$ when $x$ and $y$ are not independent. In such a case, we need to use the form $\mathrm{var}[X] = \mathrm{E}[(X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\top]$.

First of all, we start with deriving the expectation and the (co)variance of a quadratic form in general:

$$\begin{align} R(x) = x^\top A x + b^\top x + c \\ x \sim \mathcal{N}(m,\Sigma),~~ A^\top=A. \end{align}$$

$$\begin{align} \mathrm{E}[R(x)] &= \mathrm{Tr}(A\Sigma) + m^\top A m + b^\top m + c \\ &= \mathrm{Tr}(A\Sigma) + R(m) \end{align}$$

$$\begin{align} \mathrm{var}[R(x)] &= \mathrm{E}[(R(x)-\mathrm{E}[R(x)]) (R(x)-\mathrm{E}[R(x)])^\top] \\ &= \mathrm{E}[ (x^\top A x + b^\top x + \tilde{c}) (x^\top A x + b^\top x + \tilde{c})^\top ] \\ &\quad\quad ( \tilde{c} = c-\mathrm{E}[R(x)] = c - \mathrm{Tr}(A\Sigma) - R(m) ) \\ &= \mathrm{E}[ x^\top{}Ax x^\top{}Ax + 2b^\top{}x x^\top{}Ax + 2\tilde{c} x^\top{}Ax + x^\top{}b b^\top{}x + 2\tilde{c} b^\top{}x + \tilde{c}^2 ] \\ &= \dots{} \\ &= 2\mathrm{Tr}(A\Sigma A\Sigma) + 4 m^\top{}A\Sigma A m + 4b^\top{}\Sigma Am + b^\top{}\Sigma b \end{align}$$

The followings were useful for this derivation:

Now, we discuss the case 1 and 2 in the question.

Case 2.

$$\begin{align} \Delta &\sim \mathcal{N}(0,\Sigma),~~ A^\top=A. \\ R'(\Delta) &= \Delta^\top A \Delta + b^\top \Delta + c \\ \mathrm{E}[R'(\Delta)] &= \mathrm{Tr}(A\Sigma) + R(0) \\ &= \mathrm{Tr}(A\Sigma) + c \\ \mathrm{var}[R'(\Delta)] &= 2\mathrm{Tr}(A\Sigma A\Sigma) + b^\top{}\Sigma b \end{align}$$

Case 1.

$$\begin{align} x &\sim \mathcal{N}(m,\Sigma),~~ A^\top=A. \\ R(x) &= (x-m)^\top A (x-m) + b^\top (x-m) + c \\ &= x^\top A x + (b^\top - 2m^\top A) x + (m^\top{}A m - b^\top{}m + c) \\ \mathrm{E}[R(x)] &= \mathrm{Tr}(A\Sigma) + R(m) \\ &= \mathrm{Tr}(A\Sigma) + c \\ \mathrm{var}[R(x)] &= 2\mathrm{Tr}(A\Sigma A\Sigma) + 4 m^\top{}A\Sigma A m + 4(b^\top - 2m^\top A)\Sigma Am \\ &\quad + (b^\top - 2m^\top A)^\top{}\Sigma(b - 2m A) \\ &= \dots{} \\ &= 2\mathrm{Tr}(A\Sigma A\Sigma) + b^\top{}\Sigma b \end{align}$$

Therefore, the both results are the same.

$\endgroup$
  • $\begingroup$ The links you provided are priceless. Thank you! $\endgroup$ – R. Itzi Dec 29 '19 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.