# Covariance, covariance operator, and covariance function

I am trying to get my head wrapped around this article in Wikipedia. The first definition given there is the covariance of a probability measure $\mathbf{P}$:

$$\mathrm{Cov}(x, y) = \int_{H} \langle x, z \rangle \langle y, z \rangle \, \mathrm{d} \mathbf{P} (z) \tag{\ast}$$

where $x, y \in H$ (a Hilbert space). I am used to the following definition of covariance:

$$\mathrm{Cov}(X, Y) = E((X - EX)(Y - EY))$$

where $X: \Omega \to \mathbb{R}$ and $Y: \Omega \to \mathbb{R}$ are two random variables defined on the corresponding probability space $(\Omega, \mathcal{F}, \mathbf{P})$, and

$$EX = \int_{\Omega} X(\omega) \, \mathrm{d} \mathbf{P} (\omega).$$

Question 1: Can anybody please translate my definition to the one in $(\ast)$? I would like to see it rewritten in the form of $(\ast)$, and I would really appreciate some discussion to get a better understanding of what is going on there.

At the end of the article, there is a definition of the covariance function of a random element $z$:

$$\mathrm{Cov}(x, y) = \int z(x) z(y) \, \mathrm{d} \mathbf{P} (z) = E(z(x) z(y)) \tag{\ast\ast}.$$

Question 2: First of all, why don't we subtract the expected values of $z$ at $x$ and $y$? Secondly, again, I do not really see a connection with my definition. Can anybody please give a clarification?

Thank you!

Regards, Ivan

The first definition is a special case of the second. Rather than a Hilbert space H, let's look at a Banach space $X$, and distinguish it from its dual space $X^*$. Every continuous linear functional $\varphi \in X^*$ is a random variable $\varphi : X \to \mathbb R$, so it makes sense to take expectations and covariances. We define the expectation of a functional by $\mathbb E[\varphi] = \int_X \varphi[x] \, \mathrm d \mathbf P(x)$, and the covariance of two functionals to be

$$\operatorname{cov}[\psi|\varphi] = \int_X \big( \psi[x] - \mathbb E[\psi]\big) \big( \varphi[x] - \mathbb E[\varphi]\big) \, \mathrm d \mathbf P(x).$$

Now, consider a probability measure $\mathbf P$ on a Hilbert space $X = H$. By the Riesz representation theorem, we know that the dual space $H^*$ is isomorphic to $H$, and all the functionals are of the form $\varphi_h[x] := \langle h, x \rangle$.

The mean can be represented by a single element $m \in H$, which is called the "Pettis integral" of $\mathbf P$. This element satisfies the property that $\mathbb E[\varphi_h] = \varphi_h[m] = \langle h, m \rangle$ for all $h \in H$.

Consequently,

$$\operatorname{cov}[\varphi_h|\varphi_{h'}] = \int_X \big\langle h, x - m \big\rangle \big\langle h', x-m \big\rangle \, \mathrm d \mathbf P(x).$$

This is the formula you were looking for. It's just a special case of the usual covariance formula, specialized to the setting where the random variables of interest are continuous linear functionals, and the probability space is a Hilbert space.

• Thank you for your answer! One question about the concept: is the result of the expectation in $R$ or in $X$ or in $X^*$? Is this the same as the mean map or the distribution embedding to RKHS? – breezeintopl Oct 28 '15 at 4:00
• I mean, why "the mean can be represented by an element in H"? I don't understand this. "A real number(the mean) can be represented by an element in Hilbert space"? (I mean, by Riesz, functional on H can be represented. why here the mean can also be represented?) – breezeintopl Oct 28 '15 at 4:03
• Do you mean that $E[ ]: H^*\to R$ is a functional on $H^*$, so it is in $H^{**}$. And $H^{**}=H$. So then $E[]$, as element in $H^{**}$ be represented by m in $H^*$, so then $E[\phi]=<\phi,m>_{H^*}=\phi(m)=<h_{\phi},m>$? – breezeintopl Oct 28 '15 at 4:26
• It seems confusing when you write that every continuous linear functional $\varphi\in X^*$ is a random variable $\varphi:X\to\mathbb R$. $X$ is only a Banach space, it is not a probability space, right? Of course, we can use Borel $\sigma$-algebra, but what is the probability measure of this space? – Cm7F7Bb Mar 8 '16 at 10:14
• Just saw the question. I'd try to explain part(s) of your questions: In order to define a random variable $X:\Omega \to \mathbb{R}$, we don't need a probability measure on $\Omega$, we just need a sigma algebra, which in this case is the Borel one. This is because by definition, a random variable is just a measurable function, i.e. a function that pulls back the open sets of $\mathbb{R}$ to measurable subsets of $\Omega$, i.e. the subsets in the Borel sigma algebra. Next, why is the mean in $H$? Hint: it's not a real number! Think when $H=\mathbb{R}^d$, then $E(X)\in \mathbb{R}^d$ also – Mathmath Jun 24 '16 at 2:21