1
$\begingroup$

I am reading the textbook Machine Learning - A Probability Perspective, and in Chapter 8 (which talks about logistic regression) there is a paragraph that says:

Just as we prefer ridge regression to linear regression, so we should prefer MAP estimation for logistic regression to computing the MLE. In fact, regularization is important in the classification setting even if we have lots of data. To see why, suppose the data is linearly separable. In this case, the MLE is obtained when $\lVert w \lVert \rightarrow \infty$ ,corresponding to an infinitely steep sigmoid function, $I(w^Tx > w_0)$, also known as linear threshold unit. This assigns the maximal amount of probability mass to the training data. However such a solution is very brittle and will not generalize well.

Could somebody explain why the MLE for logistic regression is obtained when $ \lVert w \lVert \rightarrow \infty$ and why this assigns the maximal amount of probability mass to the training data?

$\endgroup$
  • $\begingroup$ Do you know the meaning of $w$? $\endgroup$ – caverac Jun 16 '17 at 8:46
  • $\begingroup$ w is the weight vector in logistic regression. I have edited the post to show that the discussion is about logistic regression. $\endgroup$ – lina Jun 16 '17 at 9:18
1
$\begingroup$

The key assumption here is that the data are linearly separable, meaning that there exists an $w_0$ such that $I(w_0^T x_i>0)=y_i$ for all $i=1,\cdots,n$. If this is the case, the MLE is ill-defined because $cw_0$ has higher likelihood than $w_0$ on the data if $c>1$, and hence the MLE must be $\|w_0\|\to\infty$.

However, keep in mind that MLAPP is from a firm Bayesian perspective, and as a frequentist I do not agree with this argument (that justifies the use of significant regularization for large data sets) at all :) From a frequentist perspective, if the logistic regression model is true then as the data size grows larger, it is less likely that the data are linearly separable at all. As a result, asymptotically speaking MLE is the most statistically efficient estimator in a logistic regression model with fixed dimension.

$\endgroup$
  • $\begingroup$ I see. Thanks a lot! $\endgroup$ – lina Jun 21 '17 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.