Understanding measures on the space of measures (via examples) Let $X$ be a Polish space. If it allows for an interesting answer, you may assume $X$ is compact or even $X=[0,1]$. The space $\mathcal{P}(X)$ of Borel probability measures on $X$ is also Polish (via the Prokhorov metric). Measures on $\mathcal{P}(X)$ (i.e. elements of $\mathcal{P}(\mathcal{P}(X))$) arise, for example in the ergodic decomposition. I'm looking to understand $\mathcal{P}(\mathcal{P}(X))$ better, especially through examples.
Q1. Is there a natural example of an element $\mathcal{P}(\mathcal{P}(X))$? Q.1.5 What about when $X=[0,1]$?
Q2. What results in mathematics use or refer to an element of $\mathcal{P}(\mathcal{P}(X))$? I'm aware of the ergodic decomposition and its special case, de Finetti's theorem.
Q3. Where is $\mathcal{P}(\mathcal{P}(X))$ studied? I'm aware of Billingsley's Convergence of Probability Measures and Parthasarathy's Probability Measures on Metric Spaces.
EDIT Regarding Q2, I'm most interested in classical results. The answers given by @NateEldredge and @MichaelGreinecker, while interesting and helpful, seem to regard more modern (i.e. not classical) results. I realize that 'classical' is vague, and I'll try to make what I'm after more precise if necessary. The ergodic decomposition is something I consider 'classical'.
 A: Sometimes, you want to randomly pick a point by a randomly chosen measure. The $\sigma$-algebra on $\mathcal{P}(X)$ is generated by sets of the form $$\big\{\mu\in\mathcal{P}(X):\mu(A)\leq r\big\},$$ with $A\subseteq X$ measurable and $r\in[0,1]$. Now let $\pi:X\to\mathcal{P}(Y)$ be a measurable function. Let $\mathcal{Y}$ be the $\sigma$-algebra on $Y$. Then we can define a function $$\kappa:X\times\mathcal{Y}\to [0,1]$$ by letting $\kappa(x,B)=\pi(x)(B)$ for all $x\in X$ and $B\in\mathcal{Y}$. One can rather easily show that $\kappa(x,\cdot)$ is a probability measure for all $x\in X$ and $\kappa(\cdot,B)$ is measurable for all $B\in\mathcal{Y}$. Moreover, every such function can be obtained this way. They are known as Markov kernels, probability kernels, transition probabilities, conditional probabilities, random mappings, probabilistic mappings, Young measures... they occur quite often in probability theory. Now if $\nu$ is a probability measure on $X$, we can construct a probability measure on $Y$ that corresponds to the intuitive notion of picking a point $x\in X$ according to $\nu$ and then picking a point $y\in Y$ according to $\pi(x)$. We write this measure $\nu\otimes\pi$ and have $$\nu\otimes\pi(B)=\int_X \pi(x)(B)~d\nu(x)$$ for all $B\in\mathcal{Y}$. But this can actually be seen as using $\mathcal{P}(\mathcal{P}(X))$. Using a slightly more complicated construction, we have a composition of kernels and there is actually a category with measurable spaces as objects and kernels as morphisms. If we have a whole sequence of kernels, we can use them to construct discrete time stochastic processes by the so called Ionescu-Tulcea theorem. This is the foundation of discrete time Markov processes. We can identify probability measures on $Y$ with kernels constant in the first coordinate and measurable functions $f:X\to Y$ with the kernel such that $\pi(x)$ puts probability $1$ on $f(x)$. So kernels are a very useful building block for probability theory. A good ressource on Kernels is "Foundations of Modern Probability" by Kallenberg. A book that goes into a lot of depth is "Statistical Decision Rules and Optimal Inference" by Cencov. A good starting point for the categorial viewpoint can be found here. You seem to be interested in the more topological approach. Erik Balder's material on Young measures might be useful for this (but it is hard to read).
An even more direct place where probabilities over probabilities matter is in game theory. There one studies sometimes beliefs over beliefs over beliefs over.. infinite hierarchies of beliefs. The seminal paper on the topic is this, but it is known to be really hard to read. To get the basic idea, read this. For the case of Polish spaces, this paper by Heifetz might be a good starting point (paywall warning!).
A: One example arises in the study of superprocesses.  I know only a little bit about these, so what follows may be a bit sketchy and possibly wrong.  Allison Etheridge's book An Introduction to Superprocesses looks like a promising place to read more.
For this example, let's take $X$ to be some nice compact manifold, say for instance a sphere $S^d$ or torus $(S^1)^d$, on which we know how to run Brownian motion.  ($X = \mathbb{R}^d$ might be more natural but its non-compactness complicates things slightly.)  Also, to start, I actually would rather consider positive finite measures which need not have total mass 1.  So let $\mathcal{M}(X)$ denote the space of all such measures on $X$.  Under the weak topology, this is still a nice space; it is Polish (and if I am not mistaken, even locally compact), and so its Borel $\sigma$-field gives it a nice measurable structure as well.
A typical example is branching Brownian motion.  Imagine that you start with $n$ particles in $X$, each moving independently according to Brownian motion.  However, after an exponentially distributed amount of time, a particle splits into a random number of new "offspring" particles.  Each new particle starts at the time and place where the split occurred, and evolves according to its own (conditionally) independent branching Brownian motion (and may itself split).  (One could allow the number of new particles to be zero, in which case the particle can be seen as having died.)
If we want to consider branching Brownian motion as a stochastic process $Y_t$, we need to decide in what state space it should take its values: it has to be able to represent the location of all the particles alive at a given moment, however many that happens to be.  Our first guess might be to represent the state of the process as a (finite) subset of $X$, so that our state space becomes the set of all finite subsets of $X$; let me call it $\mathcal{F}(X)$.  However, this is a bit awkward:


*

*First, we would have to decide how to define an appropriate topology and measurable structure on $\mathcal{F}(X)$, which inherits enough of the topology of $X$ that we can keep track of the fact that each particle moves continuously.  

*Second, it doesn't account for the possibility of two particles at the same location, as happens when a particle has just split, or if two particles have collided (though in this model they just pass through each other without interacting).  So maybe we need multisets or something.

*Third and most seriously, $\mathcal{F}(X)$ is not a vector space.  We cannot make sense of the expectation of $Y_t$.  Also, if we want to study the behavior as the initial number of particles goes to infinity, we will want to be able to rescale $Y_t$ somehow.
The solution to these problems is as follows: instead of representing the state of the process by a subset of $X$, we represent it by a measure, where the location of the particles are marked by unit point masses.  So the state where there are particles at locations $\{x_1, \dots, x_m\}$ corresponds to the measure $\sum_{i=1}^m \delta_{x_i}$.  We can thus think of $Y_t$ as a stochastic process taking its values in $\mathcal{M}(X)$.  This solves all the above problems: 


*

*$\mathcal{M}(X)$ has a nice natural topology and measurable structure, as discussed above.  We can also show, for example, that the process $Y_t$ is càdlàg; the jumps correspond to times when a particle splits (so if a single particle at location $x$ splits into two offspring, the process jumps from $\delta_x$ to $2 \delta_x$.

*This model naturally counts locations with multiplicity, by just putting more mass at a point if it's occupied by several particles.

*$\mathcal{M}(X)$ is (almost) a vector space (okay, it's a cone in the vector space of signed measures); we can scale and take positive linear combinations, and that's enough to handle expectations and scaling limits.
So if $Y_t$ is a process valued in $\mathcal{M}(X)$, then the law of $Y_t$ at any fixed time $t$ is a probability measure on $\mathcal{M}(X)$, i.e. an element of $\mathcal{P}(\mathcal{M}(X))$.  (The law of the entire process is a probability measure on the Skorohod space $\mathcal{D}([0, \infty), \mathcal{M}(X))$ of càdlàg paths in $\mathcal{M}(X)$, which is rather more complicated, but still a Polish space.)  We get other nice properties of $Y_t$ as well; for instance, thanks to the exponential reproduction times and independence of the offspring, $Y_t$ is Markov.
There are other models where the number of particles remains constant; in this case, we can renormalize our measures to have total mass 1, and get a process valued in $\mathcal{P}(X)$, whose one-dimensional distributions are elements of $\mathcal{P}(\mathcal{P}(X))$.  Or, we could say each particle starts with a certain amount of mass, and when it splits, that mass is divided among its offspring so that the total amount of mass in the system is conserved.  Then we can represent the state with particles at $x_1, \dots, x_m$ with respective masses $c_1, \dots, c_m$ by the measure $\sum c_i \delta_{x_i}$; if we take the total mass of the system to be 1 we again have a $\mathcal{P}(X)$ valued process.  (Here, to preserve the Markov property I guess we have to assume that if two particles collide they coalesce.)
A: 1. A category-theory point of view. $\quad$ There is little information on the space $\mathcal P(\mathcal P(X))$ that cannot be understood just by saying that $\mathcal P(\mathcal P(X))$ is the space of all probability measures on the space $Y:=\mathcal P(X)$. In particular, since $X$ is a Polish space, then $Y$ is Polish as well, thus $\mathcal P(Y)$ is in the same class as $\mathcal P(X)$, and understanding $\mathcal P(X)$ for general $X$ amounts to understanding $\mathcal P(Y)$.
In even more abstract terms, let $X,Y$ be any two completely regular Hausdorff topological spaces endowed with their Borel $\sigma$-algebras, $f\colon X\to Y$ be any continuous function, and denote by $f_*\colon \mathcal P(X)\to \mathcal P(Y)$ the map $\mu\mapsto f_*\mu$ the push-forward of $\mu\in\mathcal P(X)$ via $f$.
Then, the functor $\mathcal P$ defined on objects by $\mathcal P\colon X\mapsto \mathcal P(X)$ and on morphisms by $\mathcal P\colon f\mapsto f_*$ is an endofunctor of the category $\mathbf{crHTopB}$ of completely regular Hausdorff Borel spaces.
This functor is quite nice. For instance, it preserves Polishness or compactness (although not local compactness). That is, it remains an endofunctor of both the subcategory $\mathbf{Pol}$ of Polish spaces with continuous maps, and the subcategory $\mathbf{CPol}$ of compact Polish spaces with continuous maps.
In this generality, understanding $\mathcal P (X)$ (for Polish $X$) amounts to understanding $X$ (the general Polish space).
Vershik's towers (Башня мер). (here in Russian, p. 32ff.) The point of this level of abstraction is that it makes the construction bootstrapable.
For any Polish space $Y$, let $\delta\colon Y\to \mathcal P(Y)$ be the Dirac embedding $y\mapsto \delta_y\in \mathcal P(Y)$.
Define inductively $\mathcal P^n:= \mathcal P\circ \mathcal P^{n-1}$ with $\mathcal P^1:=\mathcal P$ the functor above. Then, $\mathcal P^n(X)$ is a Polish space, which embeds naturally into $\mathcal P^{n+1}(X)$ via the Dirac embedding $\delta^n:=\delta\colon \mathcal P^n(X)\to \mathcal P^{n+1}(X)$. The tower $(\mathcal P^n(X),\delta^n)_n$ is sometimes called Vershik's tower. It is used to generalize known facts in ergodic theory to the study of hierarchical dynamical systems, and its applications are as numerous as they are unfortunately obscure.
Metric spaces. When $(X,d)$ is a metric space, $\mathcal P$ may be extended to a functor of the category $\mathbf{Met}$ of metric spaces with continuous maps in different ways:

*

*The $L^p$-Wasserstein functor $(X,d)\mapsto \mathcal (P_p(X), W_p)$ the $L^p$-Wasserstein space with $L^p$-Wasserstein-Kantorovich-Rubinstein distance $W_p=W_{d^p}$. Note: if $(X,d)$ is unbounded, then $\mathcal P_p(X)\subsetneq \mathcal P(X)$.


*The Prokhorov functor $(X,d)\mapsto (\mathcal P(X), d_P)$, where $d_P$ is Prokhorov's distance on probability measures.
Both these functors are endofunctors of $\mathbf{CSMet}$ (complete and separable metric spaces) and of $\mathbf{CMet}$ (compact metric spaces).
The first functor, and the spaces $\mathcal P_p^2(X)$ are occasionally used in the study of bi-Lipschitz embeddings of metric spaces, snowflaking, Assouad dimension theory, etc.
1.1 Some concrete examples $\quad$
Some concrete (classes of) elements in $\mathcal P^2(X)$:
Two trivial examples.
Let $\mu\in\mathcal P(X)$. The Dirac embedding above provides two immediate examples of elements of $\mathcal P^2(X)$.

*

*The Dirac mass $\delta_\mu$ centered at $\mu$;

*The push-forward $\delta_*\mu$ of $\mu$ via the the Dirac embedding. (Note: $\delta\colon X\to \mathcal P(X)$, so $\delta_*\colon \mathcal P(X)\to \mathcal P^2(X)$, since ${}_*$ raises the index of mappings on Vershik's towers by $1$. Thus, $\delta_*\mu\in\mathcal P^2(X)$. Any $\delta_\mu$-distributed random element of $\mathcal P(X)$ is almost surely of the form $\delta_x$ with $x$ distributed as $\mu$.)

Ferguson's Dirichlet measure. Let $\mu\in\mathcal P(X)$ and $\beta>0$. An interesting element in $\mathcal P^2(X)$ is Ferguson's Dirichlet measure $\mathcal D_{\beta\mu}$.
A $\mathcal D_{\beta\mu}$-distributed random element in $\mathcal P(X)$ is almost surely of the form $\sum_{i=1}^N s_i \delta_{x_i}$, where the $x_i$'s are independent random variables identically distributed as $\mu$, and the $s_i$'s (satisfying to $s_i\geq s_{i+1}\geq 0$ and $\sum_i^N s_i=1$) are distributed according to the stick-breaking process with parameter $\beta$. If $\mathrm{supp}\mu$ is infinite, then $N=\infty$ almost surely. If otherwise $\mathrm{supp}\mu$ is finite, then $N=\#\mathrm{supp}\mu$ almost surely, and $\mathcal D_{\beta\mu}$ is the Dirichlet distribution on the standard simplex of dimension $\#\mathrm{supp}\mu-1$.
These measures have a huge number of applications to, just to mention a few ones: Bayesian non-parametrics, population genetics, statistical mechanics, representation theory, etc.,
that is impossible to exhaustively list here. (But there is a large number of monographs on the subject.)
It is possible to show (see here) that

*

*for every $\beta>0$, the maps $\mu\mapsto\mathcal D_{\beta\mu}$ define again a functor, and that the family $(\mathcal D_{\beta\mu})_{\beta>0}$ forms a sort of "topological suspension" interpolating between
$$\lim_{\beta\to 0} \mathcal D_{\beta\mu}= \delta_*\mu \qquad \text{and} \qquad \lim_{\beta\to\infty} \mathcal D_{\beta\mu}=\delta_\mu.$$


*$\mathcal P$ and $\mathcal D_{\beta\cdot}$ commute, in the following sense
$$(1) \qquad \qquad {f_*}_*\mathcal D_{\beta \mu} = \mathcal D_{\beta f_* \mu}$$
where $f\colon X\to Y$ is any measurable map, $f_*=\mathcal P^1(f) \colon \mathcal P(X)\to P(Y)$ and ${f_*}_*=\mathcal P^2(f) \colon \mathcal P^2(X)\to P^2(Y)$.
These facts have applications to the statistical-mechanics properties of some particle systems: in stat-mech terminology, $\beta>0$ is an inverse temperature, and 1. amounts to say that a $\mathcal D_{\beta\mu}$-distributed system thermalizes to $\delta_* \mu$ at infinite temperature ($\beta=0$) and christallizes to $\delta_\mu$ at $0$ temperature ($\beta=\infty$).
2. A probability-theory point of view. $\quad$ Here Nate's answer already gives the most important keyword: super-processes.
2.1 $\mathcal P(X)$-valued Markov processes $\quad$ Let me add some specific examples of $\mathcal P(X)$-valued stochastic processes $\mathbf{M}$. In all these examples, the invariant measure of $\mathbf{M}$ is a distinguished element of $\mathcal P^2(X)$ (in parenthesis).

*

*The Wasserstein diffusion on $\mathcal P(\mathbb S^1)$ (the entropic measure)


*The Modified Massive Arratia Flow and the coalescing-fragmentating Wasserstein dynamics on $\mathcal P([0,1])$ (various, including Ferguson's Dirichlet measure $\mathcal D$)


*The Dirichlet–Ferguson diffusion on $\mathcal P(M)$
with $(M,g)$ a closed Riemannian manifold (again Ferguson's Dirichlet measure $\mathcal D_{\mathrm{vol}_g}$)
2.2 Further properties of $\mathcal D$ $\quad$
Constructions related to $\mathcal D$ are also bootstrappable. This leads to hierarchical Dirichlet processes; in the language of Bayesian non-parametrics: (Dirichlet) hyper-posteriors (i.e. posteriors of posteriors) or hyper-priors, and it is closely connected to the last part of Michael Greinecker's answer about beliefs of beliefs.
3. A geometric point of view. $\quad$
When $X$ is a geometric object (e.g., a metric measure space, or a manifold), $\mathcal P(X)$ is the object of a huge interest, originally stemming from work on PDEs. This motivated the study of Wasserstein spaces as geometric objects, and in a Riemannian geometry of $\mathcal P_2(\mathbb R^n)$ (starting from here).
It is an open question whether $\mathcal P_2(M)$ ($(M,g)$ a Riemannian manifold) admits a natural volume measure related to the Wasserstein distance $W_2$. When $M$ is compact, such a volume measure (if any) would be finite, and therefore (up to normalization) a distinguished element of $\mathcal P^2(M)$ depending on the Riemannian metric $g$ of $M$. This is partly addressed here.
4. A representation-theory point of view. $\quad$
Specific elements of $\mathcal P^2(X)$ (typically, those satisfying categorical properties such as $(1)$ above), and more generally probability measures on the space of measures $\mathcal M(X)$ have some importance in the study of representation theory of large (i.e. infinite-dimensional) Lie groups.
Let me argue by example. Suppose we want to represent the group of diffeomorphisms $G=\mathrm{Diff}(M)$ of a compact manifold $M$. Clearly, $G$ acts on $M$, and therefore on the Hilbert space $L^2(M)$. However, $L^2(M)$ is by far too small for the representation to be faithful (= injective) and thus to provide useful information on the group $G$. On the other hand, $G$ acts by push-forward on the space of measures $\mathcal M(X)$. If we had a (probability measure) $\mathbb P$ on $\mathcal M(X)$, then we could represent $G$ on the Hilbert space $L^2(\mathcal M(X),\mathbb P)$ which is sufficiently large to provide useful information on $G$.
Details can be found in the introductions to this, this, and this papers. While these works are definitely recent, their understanding partly relies on a classical work  by Vershik, Gel'fand, and Graev (1975), representing $G$ on the $L^2$-space of the Poisson measure.
