1
$\begingroup$

I've come across the Naive Bayes Classifier while studying machine learning, but the trouble I'm having is with some of the probability theory used to derive the formula for finding the optimal classifier, which is why I've posted this question on mathstack(if mathstack is not the appropriate for the question, please direct me to the correct site).

My lecture slide is very unclear on how the optimal classifier is calculated: enter image description here

Given a training data set $D$ with $n$ instances, $e_1$,...,$e_n$, where each instance $e_i=(x_1=v_{i1}, x_2=v_{i2},..., x_m=v_{im})$ is seen as an attribute($x$)-value($v$) pair. And given a set of hypothesis $H$, $h_{MAP}$ is defined to the most probable hypothesis from the set $H$ given the training data set $D$, i.e the optimal classifier.
I'm not sure what the lecture slides mean by i.i.d, but I'm pretty sure that the $e_i$'s are the individual instances in the training set $D$.
Therefore, $h_{MAP}=argmax_{h_j\in H}(p(h_j|D))=argmax_{h_j\in H}(p(h_j|e_1,e_2,...,e_n))$ enter image description here I'm unsure what the bullet point is trying to say, if all hypothesis are equally likely, then $h_{MAP}$ is the maximum likelihood hypothesis, and because of this claim, the notes suddenly assume that
$p(h_j|e_1,...,e_n)=p(e_1,...,e_n|h_j)$
but, by Bayes rule:
$p(h_j|e_1,...,e_n)=\frac{p(e_1,...,e_n|h_j)p(h_j)}{p(e_1,...,e_n)}$
I haven't done much stats, so I'm sorry if the question is obvious, but how does the assumption that all hypothesis' are equally likely, lead to:
$\frac{p(h_j)}{p(e_1,...,e_n)}=1$, for all $j$
Would it always be the case that $p(e_1,...,e_n|h_j)=\Pi_{i=1}^{n}p(e_i|h_j)$, or have the notes implicitly assumed that the $e_i$ are conditionally independent. enter image description here Now the slide has assumed that each instance $e_i=(x^i, v^i)$, but each instance doesn't have a single attribute, so I'm guessing it's a bad representation of what I've assumed, i.e. $e_i=(x_1=v_{i1}, x_2=v_{i2},..., x_m=v_{im})$, unless I'm wrong, then can someone please correct me. Because the equation for $p(e_i|h_j)$ makes no sense to me.

$\endgroup$

1 Answer 1

1
$\begingroup$

The naive Bayes classifier, while appearing to be written as a conditional probability, is not in fact a conditional probability because it assumes that the data points are independent and identically distributed (i.i.d.). Bayes' theorem is used to compute the "probabilities" even though it does not really apply. These computed quantities are used as a ranking of the possibilities; higher is better. While the hypotheses are assumed equally likely, they would not be equally likely given the conditioning on the parameters.

The notation $\mathrm{arg}\ \mathrm{max}$ indicates that the left hand side is the tuple of argument values that yields the maximum value of the function.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .