Variance of a quadratic form

Question

I am considering a variance of two forms:

1. $ R(x) = (x-m)^\top A (x-m) + b^\top (x-m) + c $
2. $ R'(\Delta) = \Delta^\top A \Delta + b^\top \Delta + c $

where $x$ is a random variable of $\mathcal{N}(m,\Sigma)$ (Gaussian of mean $m$ and variance $\Sigma$). $\Delta = x-m$, so it is a random variable of $\mathcal{N}(0,\Sigma)$. $A$ is symmetric ($A^\top=A$).

Now, the two equations are the same, so $\mathrm{var}[R]$ and $\mathrm{var}[R']$ should be the same. But when I tried to derive them, the results were different. Please help me to find out this mystery.

Preliminary.

$$\begin{align} \mathrm{var}[x^\top A x] &= 2\mathrm{Tr}(A\Sigma A\Sigma) + 4 m^\top A \Sigma A m \\ \mathrm{var}[b^\top x] &= b^\top \Sigma b \end{align}$$

Case 2.

$$\begin{align} \mathrm{var}[R'] & = \mathrm{var}[\Delta^\top A \Delta] + \mathrm{var}[b^\top \Delta] \\ & = 2\mathrm{Tr}(A\Sigma A\Sigma) + b^\top \Sigma b \end{align}$$

Case 1.

$$\begin{align} R(x) &= (x-m)^\top A (x-m) + b^\top (x-m) + c \\ &= x^\top A x + (b^\top - 2m^\top A) x + \mathit{constant} \end{align}$$

$$\begin{align} \mathrm{var}[R] &= \mathrm{var}[x^\top A x] + \mathrm{var}[(b^\top - 2m^\top A) x] \\ &= 2\mathrm{Tr}(A\Sigma A\Sigma) + 4 m^\top A \Sigma A m + (b^\top - 2m^\top A) \Sigma (b - 2A m) \\ &= 2\mathrm{Tr}(A\Sigma A\Sigma) + 4 m^\top A \Sigma A m \\ &\quad + b^\top \Sigma b - 2m^\top A \Sigma b - 2 b^\top \Sigma A m + 4 m^\top A \Sigma A m \end{align}$$

Therefore, unless $4 m^\top A \Sigma A m - 2m^\top A \Sigma b - 2 b^\top \Sigma A m + 4 m^\top A \Sigma A m$ is zero, the case 1 and 2 are not the same.

Where is the problem? Thank you in advance.

Answer 1

The problem is that $\mathrm{var}[x+y] \neq \mathrm{var}[x] + \mathrm{var}[y]$ when $x$ and $y$ are not independent. In such a case, we need to use the form $\mathrm{var}[X] = \mathrm{E}[(X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\top]$.

First of all, we start with deriving the expectation and the (co)variance of a quadratic form in general:

$$\begin{align} R(x) = x^\top A x + b^\top x + c \\ x \sim \mathcal{N}(m,\Sigma),~~ A^\top=A. \end{align}$$

$$\begin{align} \mathrm{E}[R(x)] &= \mathrm{Tr}(A\Sigma) + m^\top A m + b^\top m + c \\ &= \mathrm{Tr}(A\Sigma) + R(m) \end{align}$$

$$\begin{align} \mathrm{var}[R(x)] &= \mathrm{E}[(R(x)-\mathrm{E}[R(x)]) (R(x)-\mathrm{E}[R(x)])^\top] \\ &= \mathrm{E}[ (x^\top A x + b^\top x + \tilde{c}) (x^\top A x + b^\top x + \tilde{c})^\top ] \\ &\quad\quad ( \tilde{c} = c-\mathrm{E}[R(x)] = c - \mathrm{Tr}(A\Sigma) - R(m) ) \\ &= \mathrm{E}[ x^\top{}Ax x^\top{}Ax + 2b^\top{}x x^\top{}Ax + 2\tilde{c} x^\top{}Ax + x^\top{}b b^\top{}x + 2\tilde{c} b^\top{}x + \tilde{c}^2 ] \\ &= \dots{} \\ &= 2\mathrm{Tr}(A\Sigma A\Sigma) + 4 m^\top{}A\Sigma A m + 4b^\top{}\Sigma Am + b^\top{}\Sigma b \end{align}$$

The followings were useful for this derivation:

• The Matrix Cookbook
• Matrix Reference Manual-Stochastic Matrices

Now, we discuss the case 1 and 2 in the question.

Case 2.

$$\begin{align} \Delta &\sim \mathcal{N}(0,\Sigma),~~ A^\top=A. \\ R'(\Delta) &= \Delta^\top A \Delta + b^\top \Delta + c \\ \mathrm{E}[R'(\Delta)] &= \mathrm{Tr}(A\Sigma) + R(0) \\ &= \mathrm{Tr}(A\Sigma) + c \\ \mathrm{var}[R'(\Delta)] &= 2\mathrm{Tr}(A\Sigma A\Sigma) + b^\top{}\Sigma b \end{align}$$

Case 1.

$$\begin{align} x &\sim \mathcal{N}(m,\Sigma),~~ A^\top=A. \\ R(x) &= (x-m)^\top A (x-m) + b^\top (x-m) + c \\ &= x^\top A x + (b^\top - 2m^\top A) x + (m^\top{}A m - b^\top{}m + c) \\ \mathrm{E}[R(x)] &= \mathrm{Tr}(A\Sigma) + R(m) \\ &= \mathrm{Tr}(A\Sigma) + c \\ \mathrm{var}[R(x)] &= 2\mathrm{Tr}(A\Sigma A\Sigma) + 4 m^\top{}A\Sigma A m + 4(b^\top - 2m^\top A)\Sigma Am \\ &\quad + (b^\top - 2m^\top A)^\top{}\Sigma(b - 2m A) \\ &= \dots{} \\ &= 2\mathrm{Tr}(A\Sigma A\Sigma) + b^\top{}\Sigma b \end{align}$$

Therefore, the both results are the same.