Extending a distribution over samples to a distribution over functions

Question

A gaussian process is a distribution over a function space, which can be parametrized as

\begin{align} x(\cdot) &\sim \mathrm{GP}(m(\cdot), K(\cdot,\cdot))\,. \end{align}

For simplicity, let's choose $x\colon \mathbb{R}\to\mathbb{R}$. We can write the probability density of a finite set of samples of $x$ as

\begin{align} p(\{x(l) : l \in L\}; m, K) &\propto \exp{\left[ -\frac{1}{2} \sum_{i\in L} \sum_{j\in L} \left(x(i) - m(i)\right) {K(i,j)}^{-1} \left(x(j) - m(j)\right) \right]}\,. \end{align}

(Apologies for any notational misuse here). Can this be extended to the probability density for the infinite set $x(\cdot)$? Is it valid to write

\begin{align} p(x(\cdot); m, K) &\propto \exp{\left[ -\frac{1}{2} \sum_{i\in \mathbb{R}} \sum_{j\in \mathbb{R}} \left(x(i) - m(i)\right) {K(i,j)}^{-1} \left(x(j) - m(j)\right) \right]}\,. \end{align}

It feels incorrect to me to sum over the reals rather than integrate, but I have little intuition in this area.

Answer 1

I think your notation is a little bit misleading. It is not really clear what $p(\cdots)$ is.

The definition of a gaussian process over $\mathbb{R}$ is fairly simple. All that is asked is that if you fix any finite times $t_1 < \ldots < t_n,$ then the vector $(X_{t_1}, \ldots,X_{t_n})$ is jointly gaussian.

Which, as you observe, amounts to giving a vector of means and a matrix of covariances.

Now if you look at the entire process it lives in a space of functions. You can choose many different spaces. The simplest space could be $\mathbb{R}^{\mathbb{R}}.$ But you also need a measurable space. So you can choose for example the product Borel measurable space: $(\mathbb{R}^{\mathbb{R}}, \mathcal{B}(\mathbb{R})^{\otimes \mathbb{R}} ),$ where the latter is the smallest sigma-algebra w.r.t which all the projections are measurable.

Maybe you could think that this is all too formal and not really needed. But the sigma-algebra actually restricts the collection of sets which you are allowed to measure, and - surprisingly enough - solves the problems with the notation you use in the second formula.

There is a general result that allows you to build measures on the spaces of functions by starting from the finite-dimensional distributions - which is exactly what you are asking for! This is the Kolmogorov Consistency Criterion. It tells you that if you have all the finite-dimensional distribution you can build a measure (or - the same - a random variable $X$) on the space of functions with your given finite dimensional distributions.

And now we come to your infinite sum / infinite product. It is a general fact that only countable sums or countable products can be finite (i.e. converge). This means that if you have $\sum_{i \in \mathbb{R}} a_i = a \in \mathbb{R}$ you can desume that almost all coefficients (apart from a countable set) are zero. Similarly for an uncountable positive (for the sake of simplicity) product if $\Pi_{i \in \mathbb{R}} a_i = a \in \mathbb{R}_{> 0}$ you can desume that almost all $a$ are 1.

Now let us take the probability measure associated to $X,$ and compute what you have written in the uncorrelated case $K(i,j) = \delta(i,j)$: note probability densities make sense only when integrated! Take an uncountable number of Borel sets $A_t$ and we get:

$$P(X_t \in A_t) = P(X \in \times_{t \in \mathbb{R}} A_t) = \prod\limits_{t \in \mathbb{R}}P(X_t \in A_t)$$

Now as you can see this is not well defined unless the probability is zero (not interesting for our purposes) or we take all but a countable number of $A_t$ equal to $\mathbb{R}.$

So the solution to your problem eventually is: true enough, your formula does not rigorously make sense. And it is not possible to always make sense of it. You can make sense of it when you integrate it against a set in $\mathcal{B}(\mathbb{R})^{\otimes \mathbb{R}}$ - which is by far a smaller set than the power set of $\mathbb{R}^{\mathbb{R}},$ for example because it contains only product sets where countably many elements at most can be nontrivial.