I have found the following regarding the covariance operator on Wikipedia:
For a probability measure $\mathbb{P}$ on a Banach Space $B$, the covariance of $\mathbb{P}$ is the bilinear form on the dual space $B^\ast$ given by
$$\mathrm{Cov}(x,y) = \int_B x(z)y(z) \; \mathrm{d}\mathbb{P}(z), \quad \quad x,y \in B^\ast.$$
I believe that after reading this post I can understand the motivation for defining the covariance in this way. What I still struggle with is, why is the covariance a form on the dual space? And, this being the case, what does e.g. $\mathrm{Cov}(x,x)$ even tell us about the distribution?