# Binary classification, Bayes classifier, Bayes decision boundary

I have recently come across this problem from a friend I help with stats occasionally. This however stumped me completely. I have looked online on basically every single website you can find but what I did find either I did not fully understand or I didn't fully understand well enough to explain to my friend. In my last resort I have made an account here hoping for some help.

From what I have read and understood I know the Bayes boundary is a kind of squiggly line you find which separates between classifying an observation on either end. However I don't understand how one comes to finding how to get one. Furthermore, bayesian stats is very new to me so I am struggling and therefore my friend is too. Thank you for reading and I hope someone who understands this well can explain it to me in a simple concise and easy way

Consider a binary classification problem $$Y \in \{0, 1\}$$ with one predictor $$X$$. The prior probability of being in class 0 is $$Pr(Y = 0) = \pi_0= 0.69$$ and the density function for $$X$$ in class 0 is a standard normal
$$f_0(x) = Normal(0, 1) = (1/\sqrt{2\pi})\exp(-0.5x^2).$$

The density function for $$X$$ in class 1 is also normal, but with $$\mu = 1$$ and $$\sigma^2 = 0.5$$, i.e.
$$f_1(x) = Normal(0, 1) = (1/\sqrt{\pi})\exp(-(x-1)^2).$$

(a) Plot $$\pi_0f_0(x)$$ and $$\pi_1f_1(x)$$ in the same figure.
(b) Find the Bayes decision boundary.
(c) Using Bayes classifier, classify the observation $$X = 3$$. Justify your prediction.
(d) What is the probability that an observation with $$X = 2$$ is in class 1?

Any help at all would be greatly appreciated!

• Welcome to MSE. In oder to make your equations more understandable for others, please use mathjax. I edited your post accordingly this time, but please check if it still reflects your original intend. Commented Aug 7, 2019 at 10:50
• Yes it does thank you so much! Understood, I will use mathjax for my future posts cheers Commented Aug 8, 2019 at 0:17
• Is it a homework question? Commented Aug 8, 2019 at 5:50
• I am not 100% sure but I don't think so, my friend said he got the exercise from a book he was browsing at the library but never had the answers with it so here I am! Commented Aug 8, 2019 at 12:32

Given a predictor $$X$$, you want to know from which gaussian it was most likely sampled. So you want to calculate $$\mathbb{P} (Y = 0 | X)$$ for instance. Using bayes rule, and the fact that we have access to $$\mathbb{P} (X | Y = 0)$$: $$\mathbb{P} (Y = 0 | X)= \frac{\mathbb{P} (X | Y = 0) \pi_0}{\mathbb{P}(X)}$$ and $$\mathbb{P}(X) = \pi_0f_0(x) + \pi_1f_1(x)$$ by the law of total probability.
In summary : $$\mathbb{P} (Y = 0 | X)= \frac{\pi_0 f_0(x)}{\pi_0f_0(x) + \pi_1f_1(x)}$$ $$\mathbb{P} (Y = 1 | X)= \frac{\pi_1 f_1(x)}{\pi_0f_0(x) + \pi_1f_1(x)}$$ And you're going to predict $$Y=0$$ for $$X$$ if $$\mathbb{P} (Y = 0 | X) \gt \mathbb{P} (Y = 1 | X)$$ meaning : $$\pi_0 f_0(x) \gt \pi_1 f_1(x)$$ And the decision boundary is the $$x$$ solution to: $$\pi_0 f_0(x) = \pi_1 f_1(x)$$. I'll leave the calculations to you because it's pretty basic.

The intuition behind this is that : If you have two gaussians $$G_1$$ (red) and $$G_2$$ (blue) that have same probability $$1/2$$ (like in the figure below), you're going to predict $$G_1$$ for $$X$$ if $$X \leq 0$$, and $$G_2$$ otherwise. Here the decision boundary is the intersection between the two gaussians. In a more general case where the gaussians don't have the same probability and same variance, you're going to have a decision boundary that will obviously depend on the variances, the means and the probabilities. I suggest that you plot other examples to get more intuition.

I think I'm interpreting this problem correctly, but I might also just be kidding myself.

Basic approach. Suppose that we observe $$X = x$$. We are looking for, essentially, $$P(Y = y \mid x < X < x+dx), y = 0, 1$$. Bayes permits us to rewrite this as

$$P(Y = y \mid x < X < x+dx) = \frac{P(x < X < x+dx \mid Y = y) P(Y = y)}{P(x < X < x+dx)}$$

We are given that

$$\pi_y \stackrel{\text{def}}{=} P(Y = y) =\begin{cases} 0.69 & y = 0 \\ 0.31 & y = 1 \end{cases}$$

and conveniently,

$$f_y(x)\,dx = P(x < X < x+dx \mid Y = y)$$

I would guess that the decision boundary lies wherever

$$P(Y = 0 \mid x < X < x+dx) = P(Y = 1 \mid x < X < x+dx) = \frac12$$

On the basis of the foregoing, we can rewrite this as

$$\pi_0 f_0(x) \, dx = \pi_1 f_1(x) \, dx$$

or, equivalently,

$$\pi_0 f_0(x) = \pi_1 f_1(x)$$

For this problem, I make it that the boundary consists of two values of $$x$$; between these values, one of the values of $$Y$$ is preferred; outside of those values, the other value of $$Y$$ is preferred.