Translation:Bayes Classificator -> precise math?

I want to understand the most simple form of the Bayes classificator (see here) but I want to understand it in a really precise, clean, mathematical way.

Math description of the setting:

Let us assume that $(\Omega, \mathcal{A}, P)$ is a probability space. We endow $\mathbb{R}$ with the usual Borel sigma algebra $\mathcal{B}$ and assume that $(E, \Sigma)$ is any set with sigma algebra. For a set $A \in \mathcal{B}$ and a map $X : \Omega \to \mathbb{R}$ we write $[X \in A]$ (or sometimes just $X \in A$ without the brackets) for $\{\omega \in \Omega : X(\omega) \in a\}$. Occasionally we also abbreviate $[X=a] := [X \in \{a\}]$.

Let $X_1, X_2 : \Omega \to \mathbb{R}$ and $Z : \Omega \to E$ be measurable functions. $X_1,X_2$ are called conditionally independent given $Z$ iff. for all sets $A, B \in \mathcal{B}$ and all sets $S \in \Sigma$, we have $$P( [X \in A] \cap [Y \in B] ~\mathbf{|}~ Z \in S) = P(X \in A ~\mathbf{|}~ Z \in S) \cdot P(Y \in B~\mathbf{|}~Z \in S)$$

We put $E = \{T, F\}$ with $\Sigma = \mathcal{P}(E) = \{\{\}, \{T\}, \{F\}, \{T,F\}\}$ (T stands for 'True' and 'F' for 'False'). We let $Z$ arise as the pullback to $\Omega$ of a ($\mathcal{B} \otimes \mathcal{B}-\Sigma-$) measurable map $r : \mathbb{R} \times \mathbb{R} \to \{T,F\}$ (r = 'result' that is, a function classifying emails whether they are spam or not or something like this).

* Assumption for doing the Bayes classifier: $X_1, X_2$ are conditionally independent given $Z$ *

The target of the Bayes classifier is the following: given a few values $(a_j, b_j) \in \mathbb{R} \times \mathbb{R}$ and $y_j \in \{T, F\}$ such that $r(a_j, b_j) = y_j$ for $j=1,...,n$, try to predict for every pair $(a,b) \in \mathbb{R} \times \mathbb{R}$ whether $r(a,b)=T$ or $r(a,b)=F$.

This is how people describe the setting in practice:

For a given $x=(a,b)$, compute $P(T|x)$ and $P(F|x)$ and predict '$T$' iff. the first value is bigger than the second.

I do not know what these symbols are supposed to mean but I guess that they mean the following: The first symbol means something like $P(Z=T|X=(a,b))$ with the two symbols '$Z=T$' and '$X=(a,b)$' as above (and the second one analogously with $F$).

Q1: Is this correct?

So now that the symbols make sense we want to compute

$$P(T|(a,b)) = P(Z=T|X=(a,b)) = \frac{P(Z=T \cap X=(a,b))}{P(X=(a,b))}$$

But this seems weird because whenever $X_1, X_2$ are random variables on $\mathbb{R}$ then points usually have probability zero (For example, if they are normally distributed then $P(X = (a,b)) = P_{X}(\{(a,b)\}) = P_{X_1}(\{a\}) \cdot P_{X_2}(\{b\}) = \int_{a}^a ... \cdot \int_{b}^b = 0 \cdot 0 = 0$ [note that setting $S$ to be all of $E$ we get classical independence from the conditional independence]).

Q2: Why isnt $P(X=(a,b)) = 0$ always?

Now we want to compute this value by using the theorem of Bayes:

$$P(Z=T|X=(a,b)) = \frac{P(X=(a,b)|Z=T) \cdot P(Z=T)}{P(X=(a,b))}$$ $$= \frac{P(Z=T)}{P(X=(a,b))} \cdot P(X_1=a \cap X_2=b|Z=T)$$ $$= \frac{P(Z=T)}{P(X=(a,b))} \cdot P(X_1=a|Z=T) \cdot P(X_2=b|Z=T)$$ by the conditional independence.

Ignoring the denumerator and noting that we can effectively estimate $P(Z=T)$ by computing 'nr. of T's divided by $n$', we are left with the computation of the two terms on the right.

In this article the author computes them as follows: Given a concrete set of observations $$(a_1, b_1, y_1), ..., (a_m, b_m, y_m)$$ we compute

$$\mu_{1,T} = \text{mean}(\{a_i | i \in \{1,...,n\} ~\text{is such that}~ y_i=T\})$$ $$\mu_{2,T} = \text{mean}(\{b_i | i \in \{1,...,n\} ~\text{is such that}~ y_i=T\})$$ $$\mu_{1,F} = \text{mean}(\{a_i | i \in \{1,...,n\} ~\text{is such that}~ y_i=F\})$$ $$\mu_{2,F} = \text{mean}(\{b_i | i \in \{1,...,n\} ~\text{is such that}~ y_i=F\})$$ $$\sigma_{1,T} = \sqrt{\text{var}(\{a_i | i \in \{1,...,n\} ~\text{is such that}~ y_i=T\})}$$ $$\sigma_{2,T} = \sqrt{\text{var}(\{b_i | i \in \{1,...,n\} ~\text{is such that}~ y_i=T\})}$$ $$\sigma_{1,F} = \sqrt{\text{var}(\{a_i | i \in \{1,...,n\} ~\text{is such that}~ y_i=F\})}$$ $$\sigma_{2,F} = \sqrt{\text{var}(\{b_i | i \in \{1,...,n\} ~\text{is such that}~ y_i=F\})}$$ and put $$f(x|\mu,\sigma) = \frac{1}{\sqrt{2\pi \sigma}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2} \right)$$

Then $$P(X_1=a|Z=T) = f(a|\mu_{1,T}, \sigma_{1,T})$$
$$P(X_2=b|Z=T) = f(b|\mu_{2,T}, \sigma_{2,T})$$ and so forth.

Q3: I guess this is the same issue as above: Assuming that $X_1$ is normally distributed 'on' $[Z=T]$ does not mean that single points have a propability (in particular, why is the propability of a point a value of the density function? This seems weird to me...)?

Cheers & THX in advance,

FW

• Your description is neither clear nor logical. – PSPACEhard May 29 '15 at 14:25
• Why that? Could you elaborate please? – Fabian Werner May 29 '15 at 14:32
• For example: (1) A lot of math and notations are introduced in the beginning of your description. But this seems not very helpful for your questions below. (2) Why you assume that $X_1$, $X_2$ are independent normal random variables? Your questions are about machine learning, or more specific, classifiers. The input features are usually discrete, rather than continuous. – PSPACEhard May 29 '15 at 14:42
• Did you read the question at all? What is the title of the question? The main point about the whole question is about the translation wikipedia -> precise math stuff... I do not understand your second point: dependency of variables and whether or not they are continuous are not related at all... plus: Say you have a feature birth year or so: this is not [i.e. should not treated as] discrete! Again: machine learning and discrete vs. continuous variables are unrelated... – Fabian Werner May 29 '15 at 14:54
• If I dont assume them to be conditionally independent, how should I compute the probability then? – Fabian Werner May 29 '15 at 14:58

1 Answer

In a naive Bayes classifier, you usually need to compute $P(X | C_i)$. In your context, they are $P(X | T)$ and $P(X | F)$. When $X$ is a discrete attribute, both $P(X | T)$ and $P(X | F)$ are easy to estimate based on the training data. However, if $X$ is a continuous attribute, there is a technical difficulty. As you've stated, $P(X = x | F) = P(X = x | T) = 0$. This is highly not expected. To address this problem, the trick is assume that $$P(X = x | T) = f(x | \mu_T, \sigma_T)\\ P(X = x | F) = f(x | \mu_F, \sigma_F)$$ , such that $P(X = x | T)$ and $P(X = x | F)$ won't equal to $0$. $f$ here is the same as the one in your questions. Note that $P$ here is not a valid probability any more. In fact, I don't really understand the theoretical reason for choosing Gaussian. But intuitively, if Gaussian is used, the closer the $X$ values is to the mean, the more weight is assigned (i.e., larger $P(X | T)$). This should be expected since in training data, it's very likely that most of X values are near mean, thus more weight.

EDIT:

$P(X | C_i)$, in my opinion, doesn't have to be a probability. Rather, a value representing weight is enough. When predicting the class label for a new input, we predict the label is $T$ if $\frac{P(T | X)}{P(F | X)} > 1$. Thus in fact the ratio of $\frac{P(X | T)}{P(X | F)}$ matters.

• What do you mean by 'Pr(X|T) follows a Gaussian distribution'? I guess you mean that some function $f:someSet \to \mathbb{R}$ follows a Gaussian distribution. What function f do you mean? – Fabian Werner May 29 '15 at 16:44
• The statement may be misleading. Let me edit it. – PSPACEhard May 29 '15 at 16:46
• Fixed. I think I don't clarify your questions and hope to see some other answers. – PSPACEhard May 29 '15 at 17:06
• To be honest: this answer does not make me happy. I can understand the assumption of independence as an 'accepted' and natural one... To assume, however, that this strange function $x \mapsto P(X=x|T)$ is even measurable seems very unnatural for me. – Fabian Werner May 29 '15 at 21:35
• Well. Don't regard $\Pr(X | T)$ as a probability. It is not probability any more. Instead, think it as a weight (maybe $> 1$). The more weight, the more confident that $X$ would occur. As I've stated, only the value of $\frac{\Pr(X|T)}{\Pr(X |F)}$ matters in a naive bayes classifier. So $\Pr(X |T)$ and $\Pr(X|F)$ don't need to be probability. For example, $\Pr(X = x | T) = 5$, $\Pr(X = x | F) = 3$ is equivalent to $\Pr(X = x | T) = 0.5$, $\Pr(X = x | F) = 0.3$ since $\frac{\Pr(X | T)}{\Pr(X | F)}$ are same in both cases.. – PSPACEhard May 30 '15 at 3:21