Building Bayesian Networks, Causality and Cyclic Reasoning

Question

I am studying Bayesian Statistics and I am trying to get a good understanding on Bayesian Networks, which seems to be vital in order to make something useful in Machine Learning.

Most of the texts I am reading just say simply that: "Vertices are variables and arcs are independence relations" and then build a whole bunch of complex stuff like d-separation, inference by message passing algorithms, etc. over this very superficial definition of the fundamental logic behind the Bayesian Networks.

I still did not gain an intuition about how the "independence relations" are built into a Bayesian Network, to begin with. We have a bunch of variables describing a random process, which is OK. Then we build the structure of the graph, which is where I am getting lost.

First of all, when we have the variables $V={X_1,X_2,...,X_n}$, according to the Chain Rule of Probability we have $n!$ alternatives to build the factorization of $P(X_1,X_2,...,X_n)$ like

$P(X_5)P(X_8|X_5)P(X_{12}|X_8,X_5),...$

So how can one select an ordering among these $n!$ alternatives? People talk about causality, like choosing the "causes" of an effect and making them the parents of the "effect" variable. But if we are sure that $X_i$ is the cause of $X_j$, still, for some chain rule orderings we can get a term like $P(X_i|X_j,...)$ which implies us that the "effect" $X_j$ acts incorrectly as the "cause" $X_i$. So, doesn't this conflict with the "causality" principle of a Bayesian Network?

An another issue is with the notion of the "Cyclic Reasoning". Let's think of a hypothetical machine consisting of two parallel plates which rub together. We think of three variables "Heat", "Plate Area Expansion" and "Friction". Plate Area Expansion is the effect of the cause "Heat" and "Friction" is just the effect of the plate area expansion, in turn.

This means a graph with "Heat" -> "Expansion" -> "Friction". But in reality, "Friction" is also a cause of the "Heat". We cannot draw an arc from "Friction" to "Heat" since this violates the Directed Acyclic Graph structure by creating a cycle. So, how does the Bayesian Network handle such kinds of "circularity"? What if our causality assumptions cannot be represented by a Bayesian Network, like in the above sample with "Friction" ?

Answer 1

Let's go step by step. You should distinguish the acyclic graph $G$ of network from its parameters $\theta$.

• $G$ is a graph that reflects the conditional dependencies among the variables $x_i$.
  • Parameters are the conditional probabilities encoded in the edges of $G$: if you have edges $y_1,\ldots y_n \to x$, then you have $\theta(x) = p(x \mid y_1,\ldots,y_n)$. BNs are a generative model so without $\theta$, you can not sample from them (as you do not fully define a likelihood function).

The problem of inferring a BN thus splits in two tasks: infer $G$ and $\theta$.

• For a fixed $G$, the parameters are often taken as the MLE estimator of a multinomial distribution.
• For the structure, things are harder. You usually use a score function $f(G)$ that assigns a scalar to a particular DAG, and you try to maximize it.

$f(G)$ is usually a contribution of the likelihood (computed with the parameters estimated for $G$), minus a regularization term that grows with the number of edges in $G$ and that tries to prevent overfit. Common examples are BIC or AIC regularization (they have different properties etc.).

As a general perspective on the model-selection problem you should be aware that you are dealing with a non convex state space. So, one often uses a local search strategy (e.g., hill climbing with tabu search) in the space of graphs, and allows for random restarts. The state space is huge (super-exponential), making the problem of local optima very relevant. Exact solutions are known only in particular cases where one can assume certain prior information on the model's structure.

It is worth mentioning that there are some structures (i.e., some $G$ ) that induce the same score $f$. Such models have the same ability to "explain" the data, and form something called I-equivalence class. In general, we can not expect to be able to select a model within such a class.

Concerning your causality question, that is more philosophical. BNs do not encode a notion of "time" to disantangle causality deterministically, unless in some particular case. Nonetheless, you still expect to see the dependence among the variables, which is what the edge means. The underlying assumption of BNs is that, however, you can express the causal relations as an acyclic structure; if you know that your model is circular (i.e., it has a causal feedback), then, you should not use BNs I guess.

You might want to look at the rigorous explanation in the book by Koeller and Friedman, for instance. For the structural learning part you might want to go through Heckerman's paper.