Suppose we have two classes for p=2 variate observations, where the probability for class 1 follows MVN($\mu_1$, $\Sigma$) and the population for class 2 follows MVN($\mu_2$, $\Sigma$) where $\mu_1 = \begin{pmatrix} 1 \\ 2 \end{pmatrix}$, $\mu_2 = \begin{pmatrix} 6 \\ 4 \end{pmatrix}$, and $\Sigma = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$. Assume the prior probabilities for both are the same.


Suppose we observe $x = \begin{pmatrix} 3 \\ 5 \end{pmatrix}$. Which class should x belong to given Bayes Classification Rule.

My work:

Bayes Classification Rule: $$ C_B(x) = \begin{cases} C_1 \space if\space \pi_1*f_1(x) > \pi_2*f_2(x) \\ C_2 \space if \space otherwise \end{cases}$$

Since our prior probabilities are equal and both populations have the same variance-covariance matrix I calculate the densities for both $f_1$ and $f_2$ where $$ f_i = \frac{1}{2\pi\sqrt{3}}e^{\frac{-1}{2}*(x - \mu_i)^T\Sigma^{-1}(x - \mu_i)} \space i = 1,2$$

Then $f_1 = 0.008910575$ and $f_2 = 0.001205915$. Therefore x should go be classified as class 1 by the Bayes Classification Rule.

Is it normal to see values this small, or did I do something wrong here?



1 Answer 1


As far as I can tell, your approach is correct. I wouldn't worry about the values you get - take into account that the maximum value taken by $f_i$ is $$ f_i(\mu_i) = \frac{1}{2\pi\sqrt{3}} \approx 0.0919 $$ and that the normalized distances of the point $x$ from the means $\mu_1$ and $\mu_2$ are \begin{align} \sqrt{(x-\mu_1)^T \Sigma^{-1} (x-\mu_1)} &\approx 2.1602 & \sqrt{(x-\mu_2)^T \Sigma^{-1} (x-\mu_2)} &\approx 2.9439 \end{align} which means that we are "pretty far down the bell curve."

One extra suggestion I have for you, if I may, is that it's usually more convenient to work with the log-likelihood ratio and check its sign, instead of comparing the pdfs. In particular, in this case, let \begin{align} L(x) = \log \frac{f_1(x)}{f_2(x)} &= -\frac{1}{2} (x-\mu_1)^T \Sigma^{-1} (x-\mu_1) + \frac{1}{2} (x-\mu_2)^T \Sigma^{-1} (x-\mu_2) \\ &= (\mu_1 - \mu_2)^T \Sigma^{-1} x - \frac{1}{2} \mu_1^T \Sigma^{-1} \mu_1 + \frac{1}{2} \mu_2^T \Sigma^{-1} \mu_2. \end{align} Then, following your notation, $$ C_B(x) = \begin{cases} C_1 &\text{if }L(x) > 0 \\ C_2 &\text{otherwise}. \end{cases} $$ Note that $L(x)$ is much simpler to compute than finding both $f_1(x)$ and $f_2(x)$, especially taking into account that the last two terms are constant and can be computed just once. In your case, you can easily check that $$ L\biggl(\begin{pmatrix}3\\5\end{pmatrix}\biggr) = 2 $$ and that it's indeed the same as taking $\log(f_1(x)/f_2(x))$ with the values you provide.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .