Consider a certain random variable and all its possible probability mass functions (or probability density functions).
What structure does this space have? For example, it can be endowed with a structure of a linear space just like any function of type $X \to \mathbb R\;$, but will that really make sense?
What is essential for pmf's or pdf's, what can be done with them, what are their nature, corresponding category or something?
To make things clear, I'm rather a physicist than a mathematician.