Covariance, covariance operator, and covariance function

Question

I am trying to get my head wrapped around this article in Wikipedia. The first definition given there is the covariance of a probability measure $\mathbf{P}$:

$$\mathrm{Cov}(x, y) = \int_{H} \langle x, z \rangle \langle y, z \rangle \, \mathrm{d} \mathbf{P} (z) \tag{$\ast$}$$

where $x, y \in H$ (a Hilbert space). I am used to the following definition of covariance:

$$\mathrm{Cov}(X, Y) = E((X - EX)(Y - EY))$$

where $X: \Omega \to \mathbb{R}$ and $Y: \Omega \to \mathbb{R}$ are two random variables defined on the corresponding probability space $(\Omega, \mathcal{F}, \mathbf{P})$, and

$$EX = \int_{\Omega} X(\omega) \, \mathrm{d} \mathbf{P} (\omega).$$

Question 1: Can anybody please translate my definition to the one in $(\ast)$? I would like to see it rewritten in the form of $(\ast)$, and I would really appreciate some discussion to get a better understanding of what is going on there.

At the end of the article, there is a definition of the covariance function of a random element $z$:

$$\mathrm{Cov}(x, y) = \int z(x) z(y) \, \mathrm{d} \mathbf{P} (z) = E(z(x) z(y)) \tag{$\ast\ast$}.$$

Question 2: First of all, why don't we subtract the expected values of $z$ at $x$ and $y$? Secondly, again, I do not really see a connection with my definition. Can anybody please give a clarification?

Thank you!

Regards, Ivan

Answer 1

The first definition is a special case of the second. Rather than a Hilbert space H, let's look at a Banach space $X$, and distinguish it from its dual space $X^*$. Every continuous linear functional $\varphi \in X^*$ is a random variable $\varphi : X \to \mathbb R$, so it makes sense to take expectations and covariances. We define the expectation of a functional by $\mathbb E[\varphi] = \int_X \varphi[x] \, \mathrm d \mathbf P(x)$, and the covariance of two functionals to be

$$\operatorname{cov}[\psi|\varphi] = \int_X \big( \psi[x] - \mathbb E[\psi]\big) \big( \varphi[x] - \mathbb E[\varphi]\big) \, \mathrm d \mathbf P(x).$$

Now, consider a probability measure $\mathbf P$ on a Hilbert space $X = H$. By the Riesz representation theorem, we know that the dual space $H^*$ is isomorphic to $H$, and all the functionals are of the form $\varphi_h[x] := \langle h, x \rangle$.

The mean can be represented by a single element $m \in H$, which is called the "Pettis integral" of $\mathbf P$. This element satisfies the property that $\mathbb E[\varphi_h] = \varphi_h[m] = \langle h, m \rangle$ for all $h \in H$.

Consequently,

$$\operatorname{cov}[\varphi_h|\varphi_{h'}] = \int_X \big\langle h, x - m \big\rangle \big\langle h', x-m \big\rangle \, \mathrm d \mathbf P(x).$$

This is the formula you were looking for. It's just a special case of the usual covariance formula, specialized to the setting where the random variables of interest are continuous linear functionals, and the probability space is a Hilbert space.