Let $(\Omega, \mathcal{F}, P)$ be a probability space and let $\mathbf{X}: \Omega \to \mathbb{R}^n$, $\mathbf{Y}: \Omega \to \mathbb{R}^m$ be jointly continuous random vectors. That is, there exists a $\mathcal{B}(\mathbb{R}^n \times \mathbb{R}^m)$-measurable function $f: \mathbb{R}^n \times \mathbb{R}^m \to \mathbb{R}$ such that for all $A \in \mathcal{B}(\mathbb{R}^n)$ and $B \in \mathcal{B}(\mathbb{R}^m)$, $$ P((\mathbf{X}, \mathbf{Y}) \in A \times B) = \int_A \int_B f(\mathbf{x}, \mathbf{y}) d\mathbf{y} d\mathbf{x}. $$

Let the marginal densities be denoted by $g(\mathbf{x})$ and $h(\mathbf{y})$ for $\mathbf{x} \in \mathbb{R}^n$ and $\mathbf{y} \in \mathbb{R}^m$, respectively. Then the conditional densities are given by $$ g(\mathbf{x}|\mathbf{y}) = \frac{f(\mathbf{x}, \mathbf{y})}{h(\mathbf{y})}, \qquad h(\mathbf{y}|\mathbf{x}) = \frac{f(\mathbf{x}, \mathbf{y})}{g(\mathbf{x})}, $$ from which Bayes' theorem follows, $$ g(\mathbf{x}|\mathbf{y}) = \frac{h(\mathbf{y}|\mathbf{x}) g(\mathbf{x})}{h(\mathbf{y})}. $$

Here is where I'm a bit confused. I've seen Bayes' theorem applied to estimate the parameters of a probability model, in which we specify the "prior" density of the parameters and then maximize the numerator to obtain a "maximum posterior estimate" of the parameters. Specifically, let $\theta \in \mathbb{R}^n$ denote the model parameters and $\mathbf{y} \in \mathbb{R}^m$ denote the observed data. Then this maximum posterior estimate $\hat{\theta}$ would be given by $$ \hat{\theta} = \arg \max_\theta h(\mathbf{y}|\theta) g(\theta). $$

With the above derivation of Bayes' theorem in mind, it seems clear that in order for this to make sense, both the parameters $\theta$ and the random variables $\mathbf{Y}$ must be defined as functions from the sample space to their respective codomains. This holds by definition for $\mathbf{Y}$, but it seems really strange to think of parameters as random variables. But, is this the correct viewpoint for Bayesians? Is there a more clear way to think about this?


Unknown parameters are indeed random variables according to Bayesians. You can either specify a proper prior for them (non-negative real-valued density that integrates to 1), or use an "improper prior" such as pretending that the probability of every single real number value for the parameter is equal, even though there is no actual density that does this. Often even with an improper prior, you can perform MAP estimation of the parameters given data. And by integrating over the possible parameter values (i.e. usually using a proper prior for parameters), you can get the full probability of the data under different models, so e.g. you can perform odds-ratio model comparison computations so you can see which model best fits the data, without knowing what the exact/MAP parameter values are for each model.

  • $\begingroup$ Thanks, do you have a favorite, fairly rigorous, reference book for Bayesian methods? Most of the ones I've come across assume almost no knowledge of measure theory, and I think I would develop stronger insights if I had one that assumed more. $\endgroup$ – bcf Jul 10 '15 at 20:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.