0
$\begingroup$

My intuition just led me completely wrong.

Given the fact that Gibbs sampling is working, why isn't it always possible to calculate the joint probability distribution? (Feasibility aside)

Gibbs sampling works by updating a single variable conditioned on the values of all the other variables. For that one needs to be able to calculate (for N binary variables, this makes N*2^(N-1)) conditional probabilities. I fail to get an intuitive feeling for which information are redundant in this set. And which ones are missing for calculating the joint distribution.

I know from a Bayesian point of view we are missing the marginal probabilities for the conditioned variables. And till yesterday I haven't really thought about how Gibbs sampling generates this information. Now I'm wondering. My current idea is, that this information is accessed through the fact that the current state of the Gibbs sampler always comes from the "target" distribution. However this seems to a kind of circular argument...

After doing some googleing I came up basically empty handed end ended up here. There was a question regarding the existence of a joint probability distribution given all marginals and even that seems not to be obvious. Information on Gibbs sampling more or less skip that problem and just provide proof that it works, mostly involving detailed balance, but no intuitive arguments.

So basically my questions are: 1) Given a system of N binary variables and the ability to calculate all conditionals for any single one, can I reconstruct the joint distribution? If so how? (principally not practically)

2) If not, which information is not contained in the conditionals? (All kind of cross correlations should be there, shouldn't they?)

3) Which information are redundant in the N*2^(N-1) conditional probabilities one can calculate in such a system, given that there are only 2^N-1 free parameters in such a system, and usually N>2

$\endgroup$
0
$\begingroup$

The joint probability can ALMOST be recovered directly and easily from the conditionals, i.e. all you need is just one marginal for one variable or group of variables, say $p(x_1)$. Then you have $p(x) = p(x_1)p(x_2 | x_1) p(x_3 | x_1,x_2) \ldots p(x_n | x_1, \ldots x_{n-1})$. The point of Gibbs sampling is that you DON'T know how to sample from the joint, even to get one initial sample, so you start at an arbitrary $x$ in the support of the distribution and then repeatedly do Gibbs sampling steps. Since the conditional probabilities can generate a sample, at least if your support is open and connected, they do in principle define the joint distribution, but technically by following Gibbs sampling directly it's in terms of messy nested limits and integrals that go on forever, since the initial point will have at least a small effect on the sample you get, unless you take the limit as the number of Gibbs steps for a sample goes to $\infty$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.