# why do we need regularization when there is a lot of data

I am reading the textbook Machine Learning - A Probability Perspective, and in Chapter 8 (which talks about logistic regression) there is a paragraph that says:

Just as we prefer ridge regression to linear regression, so we should prefer MAP estimation for logistic regression to computing the MLE. In fact, regularization is important in the classification setting even if we have lots of data. To see why, suppose the data is linearly separable. In this case, the MLE is obtained when $\lVert w \lVert \rightarrow \infty$ ,corresponding to an infinitely steep sigmoid function, $I(w^Tx > w_0)$, also known as linear threshold unit. This assigns the maximal amount of probability mass to the training data. However such a solution is very brittle and will not generalize well.

Could somebody explain why the MLE for logistic regression is obtained when $\lVert w \lVert \rightarrow \infty$ and why this assigns the maximal amount of probability mass to the training data?

• Do you know the meaning of $w$? – caverac Jun 16 '17 at 8:46
• w is the weight vector in logistic regression. I have edited the post to show that the discussion is about logistic regression. – lina Jun 16 '17 at 9:18

The key assumption here is that the data are linearly separable, meaning that there exists an $w_0$ such that $I(w_0^T x_i>0)=y_i$ for all $i=1,\cdots,n$. If this is the case, the MLE is ill-defined because $cw_0$ has higher likelihood than $w_0$ on the data if $c>1$, and hence the MLE must be $\|w_0\|\to\infty$.