Bayesian Logistic Regression, conditional probability integration

In Andrew NG's Lectures (CS229), the Bayesian Logistic Regression section contained a formula;

$$P(Y|X,S)=\int_\theta P(Y|X,\theta)P(\theta|S)d\theta$$

Here, $\theta$ is treated as a random variable. $S$ is the set of points ${[X^{(i)},Y^{(i)}]_{i=1}^{m}}$.

Using conditional probabilities, it does make intuitive sense although I would really appreciate a rigorous proof of the equation.

From what I got: $$P(Y|X,S)=\int_\theta P(Y,\theta|X,S)d\theta$$ $$= \int_\theta P(Y|\theta,X,S)P(\theta|X,S)d\theta$$

Does it assume any sort of independence?

Again, I get the intuition, but I can't seem to arrive at the final answer. A written out proof or required equations to prove the result would be appreciated.

• What are $X$ and $Y$? Discrete or continuous random variables (or vectors)? Please include as much context as possible directly in the question instead of pointing to some course. – paf Jul 7 '18 at 7:21
• @paf In the context of this problem, X(i) and Y(i) are points in the sample space, S. X(i) is an n dimensional vector and Y(i) is the corresponding classification of that point. X and Y themselves are continuous random variables. – Utsav Dutta Jul 7 '18 at 7:26
• Treat X as a new point in the hyperspace and P(Y=y|X) as the probability of classifying the new point X as Y=y. – Utsav Dutta Jul 7 '18 at 7:27
• – Henry Jul 7 '18 at 8:17
• What does $\int_\theta \ldots d\theta$ means (where $\theta$ is a r.v.)? – d.k.o. Jul 7 '18 at 19:24

The assumption applied is that of conditional independence. If $$Z_1$$ and $$Z_2$$ are independent then $$p(Z_1\lvert Z_2) = P(Z_1)$$ hence $$Z_2$$ can be removed from the set of variables on which one is conditioning. Similarly with conditional independence if $$Z_1$$ and $$Z_3$$ are conditionally independent given $$Z_2$$ then $$p(Z_1\lvert Z_2,Z_3) = p(Z_1\lvert Z_2)$$ so ones $$Z_2$$ is "controlled for" $$Z_1$$ does not depend on $$Z_3$$.
$$p(Y\lvert \theta , X, S) = p(Y\lvert \theta ,X)$$
the dependent variable $$Y$$ only depends on $$S$$ through $$X$$ and $$\theta$$
$$p(\theta \lvert X,S) = p(\theta \lvert S)$$
so parameters $$\theta$$ do not depend on $$X$$ once $$S$$ is given.