I am studying Bayesian Statistics and I am trying to get a good understanding on Bayesian Networks, which seems to be vital in order to make something useful in Machine Learning.
Most of the texts I am reading just say simply that: "Vertices are variables and arcs are independence relations" and then build a whole bunch of complex stuff like d-separation, inference by message passing algorithms, etc. over this very superficial definition of the fundamental logic behind the Bayesian Networks.
I still did not gain an intuition about how the "independence relations" are built into a Bayesian Network, to begin with. We have a bunch of variables describing a random process, which is OK. Then we build the structure of the graph, which is where I am getting lost.
First of all, when we have the variables $V={X_1,X_2,...,X_n}$, according to the Chain Rule of Probability we have $n!$ alternatives to build the factorization of $P(X_1,X_2,...,X_n)$ like
$P(X_5)P(X_8|X_5)P(X_{12}|X_8,X_5),...$
So how can one select an ordering among these $n!$ alternatives? People talk about causality, like choosing the "causes" of an effect and making them the parents of the "effect" variable. But if we are sure that $X_i$ is the cause of $X_j$, still, for some chain rule orderings we can get a term like $P(X_i|X_j,...)$ which implies us that the "effect" $X_j$ acts incorrectly as the "cause" $X_i$. So, doesn't this conflict with the "causality" principle of a Bayesian Network?
An another issue is with the notion of the "Cyclic Reasoning". Let's think of a hypothetical machine consisting of two parallel plates which rub together. We think of three variables "Heat", "Plate Area Expansion" and "Friction". Plate Area Expansion is the effect of the cause "Heat" and "Friction" is just the effect of the plate area expansion, in turn.
This means a graph with "Heat" -> "Expansion" -> "Friction". But in reality, "Friction" is also a cause of the "Heat". We cannot draw an arc from "Friction" to "Heat" since this violates the Directed Acyclic Graph structure by creating a cycle. So, how does the Bayesian Network handle such kinds of "circularity"? What if our causality assumptions cannot be represented by a Bayesian Network, like in the above sample with "Friction" ?