Can anyone point me to literature on Bayesian learning when the new information has the form “If A, then B”? I’m familiar with the rule that after one learns X, posterior probability P(Y) equals prior conditional probability P(Y|X). But what about cases where X is itself a conditional statement?

For context:

  1. I know about updating on less-than-certain information (Jeffrey conditioning). That doesn't seem like the key here, however.
  2. I realize one can treat learning “If A then B” as learning “Either B or else not-A.” But is that the only option? Has anyone treated learning "If A, then B" as a direct prompt to revise one’s conditional probability P(B|A)?
  3. The textbook answer may be “That can’t be done, because there’s no formalism for a conditional probability P(B|A) itself being conditioned on a third proposition C (where this could be “If A then B”)." If that’s the answer, fine. But I’d also like to know if the range of options includes, “Oh, yes, that double-conditional thing sounds like a problem, but really it’s not, (etc.).”
  4. Please, if possible, give one or more specific references (texts/journal articles), or in lieu of that, a technical phrase I can use in a Google search.

I can give you the answer, which I think is quite simple once you know it, but I'm afraid I don't really have any references for it. I learnt from E.T. Jaynes' "Probability Theory: The Logic of Science", but that spends a lot of its time on pro-Bayesian propoganda, and so might not be the best textbook.

Anyway the answer is sort of "just do it", but I'll explain the calculation since juggling expressions like "$X\wedge(B\Rightarrow A)$" is very confusing until you get used to it.

(Notation: I'll use "$¬$" for "not", "$\wedge$" for "and", and "$\vee$" for "or".)

The definition of conditional probability is $$P(Y|X)=\frac{P(Y\wedge X)}{P(X)}$$ So when we want to calculate $P(Y|(A\Rightarrow B))$ we just use the above formula with $(A\Rightarrow B)$ replacing $X$ to get $$P(Y|(A\Rightarrow B))=\frac{P(Y\wedge (A\Rightarrow B))}{P(A\Rightarrow B)}$$ The thing on the left hand side is what you want. The probability expressions on the right hand side are just unconditional probabilities, so you should be able to calculate them straight away (e.g. by the usual method of seeing what proportion of possible outcomes satisfy the statement).

(The above should answer your question, but since you mentioned "double conditioning", I thought I'd also explain how that works:

If we've already conditioned on $W$ and we want to futher condition on $X$ then we just repeat the definition but with "$|W$" to the right of everything: $$P(Y|X,W)=\frac{P(Y\wedge X|W)}{P(X|W)}$$ Note that we write $P(Y|X,W)$ for the probability of $Y$ conditioned on $W$ and then $X$, rather than $P(Y|X|W)$ like you might expect. You'll see why this is in a second.

We think of conditioning on $W$ as "restricting to the set of possibilities where $W$ is true". So conditioning on $W$ and then $X$ is restricting to the set of possibilities where $W$ is true and furthermore $X$ is true. I.e. we are restricting to the set of possibilities where $X$ and $W$ are both true. So in fact the three following expressions are all equal. $$P(Y|X,W)=P(Y|W,X)=P(Y|X\wedge W)$$ You can check this by expanding them out using the definitions of conditional probability above. They all give $$\frac{P(Y\wedge X\wedge W)}{P(X\wedge W)}.$$ So in fact people don't ever do "double conditioning" they just think of "conditioning on $X$ and on $W$" as a synonym of "conditioning on $X\wedge W$". This explains why we use the notation $P(Y|X,W)$; it's just the thing we want the probability of, followed by the "$|$" symbol, followed by the list of the things we are conditioning on.)


@Oscar: I hope I'm doing this right. My reply is apparently too long for a comment.

Thank you for that detailed answer. I've seen Jaynes’s name a lot lately, and I appreciate your tip that here, too, he might have something interesting to say.

I also appreciate the notes on double conditioning, especially the part about the comma notation. However, I think I caused some confusion. What I ended up saying in my earlier context note 3 is that maybe there’s no formalism for $P((B|A)|C)$. If I understand you right, you’re pointing out that this can be treated as $P(B|(A\wedge C))$, which I get.

However, what I meant to say was that maybe there’s no formalism for evaluating $P(C|(B|A))$. Which brings us to your main reply. When you say that $P(Y\wedge (A\Rightarrow B))$ and $P(A\Rightarrow B)$ are just unconditional probabilities, I think you must have in mind that $\Rightarrow $ is the material conditional; that is, $A\Rightarrow B$ means the same thing as $¬A\vee B$. If you don’t mean that, then I’m not understanding you. But if you do mean that, then this is what I was trying to anticipate in my context note 2. I know one can construe conditioning on "If $A$, then $B$" as conditioning on the material conditional, but I was wondering whether it was the only way to go. Your answer/Jaynes's answer would seem to be that yes, this is the only way to go!

  • $\begingroup$ You've understood me correctly, and that is the only way to go. One thing that confused me and might be confusing you is that the expression "$A|B$" doesn't represent an event in the way that "$A\wedge B$" or "$A\Rightarrow B$" does. When we write $P(A|B)$ there's no way of thinking of this as the probability of some event "$A|B$". So saying $P(C|(B|A))$ doesn't make sense. What I wrote, $P((B|A)|C)$, doesn't make sense either unless it's just a shorthand for $P(B|A,C)$. I think of the bar "$|$" as separating the two arguments of the "conditional probability" $P(\centerdot|\centerdot)$. $\endgroup$ – Oscar Cunningham May 23 '14 at 9:28
  • $\begingroup$ Where, as we've seen, the "second argument" could actually be a list of more than one event. $\endgroup$ – Oscar Cunningham May 23 '14 at 9:31
  • $\begingroup$ "When we write $P(A|B)$ there's no way of thinking of this as the probability of some event "$A|B$"." Yup, that’s the orthodox view, and I’m not going to challenge it. However, there’s a body of literature about conditional events. See the exchange on conditional events and the linked Wikipedia article. $\endgroup$ – StumpyLeg May 23 '14 at 14:30
  • $\begingroup$ Ah, I understand your question a bit better now. You know everything I said above, but you want to know if there are any other ways of dealing with the information "if $A$ then $B$". But surely any such way would either give the same answer (in which case it's not really different) or give a different answer (in which case it's wrong). Jaynes often belabours the point that Bayesian probability is consistent. If there's more than one way to do something then either way should give the same answer. $\endgroup$ – Oscar Cunningham May 23 '14 at 15:14
  • $\begingroup$ "You know everything I said above, but you want to know if there are any other ways of dealing with the information "if $A$ then $B$"." Yes, that's basically where I was coming from. "But surely any such way would either give the same answer." Well, maybe, but maybe not. To have a high degree of confidence that if $A$, then $B$ is, plausibly, to have a high $P(B|A)$, which isn't the same thing has having a high $P(¬A \vee B)$. So if one updates one's probability function to satisfy the former condition, that's not necessarily the same thing as updating it to satisfy the latter condition. $\endgroup$ – StumpyLeg May 23 '14 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.