I have a single regressor $X$ and response $y$, where $y=n_k/n$ if $X$ is of class $k$, and $k=1,2$. Let $n_k$ denote the number of observations in class $k$, and $n$ the total number of observations. Denote $\mu_k$ as the mean of class $k$, $\Sigma$ the covariance matrix of X, and $\pi_k$ the probability of class $k$ occurring. I have the general discriminant of $\delta(x)=x'\Sigma^{-1}\mu_k-\frac{1}{2}\mu_k'\Sigma^{-1}\mu_k+\log(\pi_k)$,

and I know the decision boundary between classes 1 and 2 is given by $$\log(n_2/n)-\frac{1}{2}\mu_2'\Sigma^{-1}\mu_2+x'\Sigma^{-1}\mu_2=\log(n_1/n)-\frac{1}{2}\mu_1'\Sigma^{-1}\mu_1+x'\Sigma^{-1}\mu_1$$

I let $\hat\sigma^2$ estimate $\Sigma$ and $\hat\mu_k$ estimate $\mu_k$, and then by treating these as scalars and using algebraic manipulation, I obtain


For any $X$ that doesn't satisfy this equation, I suspect that determining which side of this equation is greater for a given $X$ will tell me which class $X$ belongs to, but I can't figure out which side would correspond to which class (if this is a correct assumption).


When we are not sure which side does a classifier assign a data point to, a possible way is to substitute some points that we know it belongs to a particular class to observe the outcome. We can try out with $\hat{\mu}_1$ and $\hat{\mu}_2$ where $\hat{\mu}_1\ne\hat{\mu}_2$

The boundary is


Let $$f(x) = \frac{(\hat\mu_2-\hat\mu_1)x}{\hat\sigma^2}-\frac{(\hat\mu^2_2-\hat\mu^2_1)}{2\hat\sigma^2}=\frac{(\hat{\mu}_2-\hat{\mu}_1)}{\hat{\sigma}^2}\left( x-\frac{\hat{\mu}_1 + \hat{\mu}_2}2\right).$$

$$f(\hat{\mu}_1) =\frac{(\hat{\mu}_2-\hat{\mu}_1)}{\hat{\sigma}^2}\left( \hat{\mu}_1-\frac{\hat{\mu}_1 + \hat{\mu}_2}2\right)=\frac{-(\hat{\mu}_2 - \hat{\mu}_1)^2}{2\hat{\sigma}^2} <0$$

$$f(\hat{\mu}_2) =\frac{(\hat{\mu}_2-\hat{\mu}_1)}{\hat{\sigma}^2}\left( \hat{\mu}_2-\frac{\hat{\mu}_1 + \hat{\mu}_2}2\right)=\frac{(\hat{\mu}_2 - \hat{\mu}_1)^2}{2\hat{\sigma}^2} >0$$

Hence if $f(x) < -\log(n_2/n_1)$, it belongs to the first class.

If $f(x) > -\log(n_2/n_1)$, it belongs to the second class.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.