My intuition just led me completely wrong.
Given the fact that Gibbs sampling is working, why isn't it always possible to calculate the joint probability distribution? (Feasibility aside)
Gibbs sampling works by updating a single variable conditioned on the values of all the other variables. For that one needs to be able to calculate (for N binary variables, this makes N*2^(N-1)) conditional probabilities. I fail to get an intuitive feeling for which information are redundant in this set. And which ones are missing for calculating the joint distribution.
I know from a Bayesian point of view we are missing the marginal probabilities for the conditioned variables. And till yesterday I haven't really thought about how Gibbs sampling generates this information. Now I'm wondering. My current idea is, that this information is accessed through the fact that the current state of the Gibbs sampler always comes from the "target" distribution. However this seems to a kind of circular argument...
After doing some googleing I came up basically empty handed end ended up here. There was a question regarding the existence of a joint probability distribution given all marginals and even that seems not to be obvious. Information on Gibbs sampling more or less skip that problem and just provide proof that it works, mostly involving detailed balance, but no intuitive arguments.
So basically my questions are: 1) Given a system of N binary variables and the ability to calculate all conditionals for any single one, can I reconstruct the joint distribution? If so how? (principally not practically)
2) If not, which information is not contained in the conditionals? (All kind of cross correlations should be there, shouldn't they?)
3) Which information are redundant in the N*2^(N-1) conditional probabilities one can calculate in such a system, given that there are only 2^N-1 free parameters in such a system, and usually N>2