Gaussian Class-Conditional distribution

(a) Consider a binary classifier (i.e. y ∈ {0, 1}). Suppose the class conditional distribution, $p(x^{(n)}|y^{(n)} = j)$ is a Gaussian with mean $µ_{j}$ and covariance matrix $Σ_{m}$, for j = 1, 2. Let δ be the decision boundary between two classes, i.e.

δ(x) = $\log\frac{p(y = 1|x)}{p(y = 2|x)}$

Further suppose that the prior over y is uniform. Write the Bayes classifier as y = f(x) = sign(δ(X)) and simplify δ as much as possible. What is the geometric shape of the decision boundary?

(b) Repeat (a) but assume that the two Gaussians have identical covariance matrices. What is the geometric shape of the decision boundary?

(c) Repeat (a) but assume now that the two Gaussians have covariance matrix which is equal to the identity matrix ($Σ_{1} = Σ_{2} = I$). What is the geometric shape of the decision boundary?

Note: Recall that the p-dimensional Gaussian distribution has the form

N(x; µ, Σ) = $\frac{1}{(2π)^{p/2} |Σ|^{1/2}}$ $e^{-\frac{1}{2}(x − µ)^{T} Σ^{-1}(x − µ)}$


closed as off-topic by qwr, choco_addicted, NCh, André 3000, John Bentin Mar 30 '18 at 6:23

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question is missing context or other details: Please improve the question by providing additional context, which ideally includes your thoughts on the problem and any attempts you have made to solve it. This information helps others identify where you have difficulties and helps them write answers appropriate to your experience level." – qwr, choco_addicted, NCh, André 3000, John Bentin
If this question can be reworded to fit the rules in the help center, please edit the question.


EDIT: made edit to the formulas because I forgot the log normalizing constants

The decision boundary is when one of the two possible Gaussians is more likely than the other. You can see this in the formula $\log\frac{P[y=1|x]}{P[y=2|x]}$.

The log will be negative if $P[y=1|x]<P[y=2|x]$ and nonnegative when $P[y=1|x] \geq P[y=2|x]$.

The decision boundary is when they are equal, giving $\delta(x) = 0$.

Taking the log of the ratio of the two densities gives us:

\begin{equation} \begin{split} & \log\frac{P[y=1|x]}{P[y=2|x]} = 0 \\ \Leftrightarrow & (x-\mu_1)\Sigma^{-1}_1(x-\mu_1) - (x-\mu_2)\Sigma^{-1}_2(x-\mu_2) + c = 0 \\ \Leftrightarrow & \sum_{i=1}^p(x-\mu_{i_1})^2\lambda_{ii_1} + \sum_{i\neq j}(x-\mu_{i_1})(x-\mu_{j_1})\lambda_{ij_1} - \sum_{i=1}^p(x-\mu_{i_2})^2\lambda_{ii_2} - \sum_{i\neq j}(x-\mu_{i_2})(x-\mu_{j_2})\lambda_{ij_2} + c = 0 \end{split} \end{equation}

Where $\Lambda = \Sigma^{-1}$ and $\lambda_{ij}$ is the $i,j$ element of $\Lambda$, and $c$ is a constant representing the log of the normalizing constants: $c = \frac{1}{2}\log|\Sigma_1| - \frac{1}{2}\log|\Sigma_2|$.

Note that this is the difference of two quadratic polynomials in $x$, which is also a quadratic polynomial in $x$. So the shape of the decision boundary is the set of solutions to this polynomial.

You should be able to use that formula to solve the last two questions by adjusting the parameters.

  • $\begingroup$ How did you get from the log ratio to the (x - mean)^T (covariance matrix inverse) (x - mean) - ... = 0 line? Why can you ignore the log(2pi), the log |covariance matrix|_1, log |covariance matrix|_2 and the log(P(y = 1)), log(P(y = 2)) terms? $\endgroup$ – MadDog Mar 30 '18 at 16:31
  • $\begingroup$ If you take the log of the densities the exponentials go away, and you get the polynomial arguments of the exponential function. The rest of the terms fall out with the logarithm, but none of them involve the $x$ variable, so you can think of them as just a constant added to the quadratic polynomial. $\endgroup$ – Ryan Warnick Mar 31 '18 at 2:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.