I have read other materials on Bayesian statistics but am using the Wikipedia page on Bayesian inference to frame this question.

I have a working understanding of the $$P(H \mid E) = \frac{P(E \mid H) P(H)}{P(E)}$$ version of Bayesian inference, and can do examples/exercises with that method. But as soon as people shift to $$P(\theta \mid X, \alpha) = \frac{P(X \mid \theta) P(\theta \mid \alpha)}{P(X \mid \alpha)}$$ I get confused. I know that they are describing the same theorem, and I have a basic understanding of probability distributions and their parameters, but this still is not clicking for me. Could someone either help me conceptually or give a worked example that might help me understand the formal definition? I'm trying to reach an intuitive understanding of this. I have done a lot of reading and am still extremely confused.

Thank you!


Just work with the discrete setting first. All that is said is just the following. You suspect that some random variable $X$ (say, with values $0,1$) may have several possible distribution laws (say, $P(0)=\frac 13, P(1)=\frac 23$ (1), $P(0)=\frac 12, P(1)=\frac 12$ (2), and $P(0)=\frac 23, P(1)=\frac 13$ (3). So your $\theta\in\{1,2,3\}$ is just the choice of this law. You want to figure out which law is the case. All you can do is to make an initial guess (say, you suspect that (1) is more likely than the other two, which are equally likely. So, you make an educated guess and bet that there is about $1/2$ chance that you are dealing with law (1) and assign $\frac 14$ chance to each of the other $2$.

Now you observe $X$ and it comes out as $1$, say. So you look at how this changes your opinion about the likelihood of each law. Remember that your model of the generation of $X$ is a two-step one now: first you choose randomly one of 3 probability laws and then choose $X$ according to that law.

The full probability ($P(X|\alpha)$) of getting $X=1$ is then $\frac 12\frac 23+\frac 14\frac 12+\frac 14\frac 13=\frac{13}{24}$. Now you just look at the part each law contributed to this whole probability, which is given by the corresponding individual product of fractions $P(X|\theta)P(\theta|\alpha)$ (I wrote the factors in the reverse order above). Those parts are $\frac 13=\frac{8}{24}, \frac 18=\frac 2{24}, \frac 1{12}=\frac 2{24}$. Now just compute what portion each part makes of the whole $\frac{13}{24}$. You'll get $\frac 8{13},\frac 3{13}, \frac 2{13}$ correspondingly. Thus your belief in law (1) got reinforced while your beliefs in two other laws got diminished (which is not really surprising). The next sample will create another update, and so on.

Just keep in mind that what you are trying to evaluate here is not the likelihood of one event given another event (like the probability of rain given that the sky is cloudy) but the likelihood of the distribution given an event (like what is the chance that, given that the rain fell, it is the first weather station that predicts the weather correctly if it was saying that it would rain with probability 20% and the second one put that chance at 40%, assuming that one of them is always correct and the other one is taking its data from the ceiling but you don't know which one is which). BTW, you can try this example yourself now. All you need is to assign some prior probabilities to the 2 weather stations and see how your opinions of them get updated after you get wet when walking outside.

  • $\begingroup$ This is fantastic! Could I just ask how hyperparameters might fit into that example? $\endgroup$ – JohnDoeVsJoeSchmoe May 30 '18 at 5:38
  • $\begingroup$ @JohnDoeVsJoeSchmoe Have you tried to read en.wikipedia.org/wiki/Conjugate_prior ? The word "hyperparameter" just describes a reasonable set of initial guesses you are willing to consider at all. In the discrete setting it is often just the set of all feasible distributions. For instance, in the weather station example, instead of actually making an initial guess and proceeding with numbers, you can just say that you consider the family of all initial guesses $P(S1\text{ tells truth})=\alpha\in[0,1]$, proceed symbolically, and get a ready recipe for any prior one can choose. $\endgroup$ – fedja May 30 '18 at 5:59
  • $\begingroup$ @@JohnDoeVsJoeSchmoe In the continuous case (or even in the discrete case of large size), considering all possible priors may be a headache, so you choose a decent family of priors (say, Gaussians with parameters $(\mu,\sigma)$ if possible laws for $X$ are described by one real parameter ) and run the symbolic computations only for those priors. In general, the updated laws may fail to belong to the same family. However in the case of conjugate priors, they will, so you may pretend that instead of updating the full distribution laws for old $\mu,\sigma$, you update $\mu,\sigma$ themselves. $\endgroup$ – fedja May 30 '18 at 6:07
  • $\begingroup$ @@fedja I have a question about the third paragraph of your initial response. "The full probability $(P(X|α))$ of getting $X=1$ is then... $\frac{13}{24}$." How would you describe how you calculated that? Maybe something like "weighted marginal distribution?" Does that formula have a name? Also, could you explain what you did next? I don't understand how you calculated $P(X = 1 | \theta) P(\theta)$. $\endgroup$ – JohnDoeVsJoeSchmoe May 31 '18 at 3:01
  • $\begingroup$ @JohnDoeVsJoeSchmoe "Full probability formula" or something like that. As to $P(X=1|\theta)P(\theta)$, let's say you are interested in distribution law (1). Then $P(\theta)=\frac 12$ (the probability assigned to it by your prior) and $P(X=1|\theta)=\frac 23$ (because that is the probability $\theta$ assigns to $1$, so you just multiply them out). $\endgroup$ – fedja May 31 '18 at 11:20

$\def\Ω{{\mit Ω}}\def\Θ{{\mit Θ}}\def\peq{\mathrel{\phantom{=}}{}}$Here is a detailed derivation of $p(θ \mid X, α) = \dfrac{p(X \mid θ) p(θ \mid α)}{p(X \mid α)}$.

Since $α$ is a hyperparameter, it is more rigorous to write the identity above as$$ p_α(θ \mid X) = \frac{p_α(X \mid θ) p_α(θ)}{p_α(X)}. $$ Now, suppose $X$ is a random variable on $(\Ω, \mathscr{F})$ and $θ$ on $(\Θ, \mathscr{G})$. For any $A \in \mathscr{F}$ and $B \in \mathscr{G}$, by the definition of conditional probability,\begin{align*} &\peq \frac{P_α(X \in A \mid θ \in B) P_α(θ \in B)}{P_α(X \in A)} = \frac{P_α(X \in A,\ θ \in B)}{P_α(θ \in B)} · \frac{P_α(θ \in B)}{P_α(X \in A)}\\ &= \frac{P_α(X \in A,\ θ \in B)}{P_α(X \in A)} = P_α(θ \in B \mid X \in A). \end{align*} Therefore, the distributions satisfy$$ \frac{p_α(X \mid θ) p_α(θ)}{p_α(X)} = p_α(θ \mid X). $$

  • $\begingroup$ I'm sorry, my "not enough detail" request wasn't very clear. I'd really like something helping my intuition bridge the gap between the "basic" version of the formula, and the fancier one with the parameters. Less math detail and more English detail. Maybe an example, accompanied by a description showing exactly how the parameters fit in. I'm very grateful for the time you put into this though! $\endgroup$ – JohnDoeVsJoeSchmoe May 27 '18 at 20:35
  • $\begingroup$ @JohnDoeVsJoeSchmoe Well, $p_α(θ\mid X)=\dfrac{p_α(X\mid θ) p_α(θ)}{p_α(X)}$ is indeed the basic version. $\endgroup$ – Saad May 28 '18 at 1:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.