# Why Logistic Regression for Classification Problems?

In a class on machine learning, we covered classification problems. In such a problem, you are studying a property of some object, say malignity of tumors in a patient. You are first given a training set, which consists of a set of ordered pairs $(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), ... (x^{(n)}, y^{(n)})$, where each $x^{(i)}$ is a vector of desired parameters (in the tumor example, size could be such a parameter), while each $y^{(i)}$ is either zero or one depending on the status of this property (in the tumor example, one might say a tumor is benign if $y^{(i)} = 0$, and malign if $y^{(i)}=1$. By fitting a function to this training set, one finds a hypothesis function, which hopefully predicts whether tumors are benign or malign in the future. But how to fit such a function? The professor of the class gave an example that wouldn't work, linear regression. Linear regression was shown as a poor method because outliers in the training set would influence the hypothesis too drastically. Then the professor said that the better method for classification problems was logistic regression. However, he did not explain this -- from an observer's point of view, it seemed logistic regression was chosen ad hoc as a fitting method. Could someone please ex

## 2 Answers

Maybe your professor should ex, since he's the one who introduced the logistic regression? One (possibly weak) reason to use the logistic curve is: it is one of the simplest functions one can think of that increases from 0 to 1, always staying between 0 and 1. Thus, its value at $x$ can be interpreted as the likelihood of $x$ belonging to the "$y=1$" group.

Consider a simple example: Suppose we have object's temperatures, and two classes, hot and cold. If that's all, at the end of the day you want a solution that classifies the objects to hot or cold. Ideally there would exist a cut-off temperature T=$X_0$, and all objects less than $X_0$ would be cold, while all with greater hot.

In this case you need a "step function", that is $$f(x)= \{\begin{array}{l} 0,x<X_0 \\ 1,x\geq X_0\end{array}$$

The problems with this function are: 1. that it is not-continuous (so the derivative is not defined everywhere, and this makes math difficult) and 2. that it does not allow for situations where we are a bit uncertain (because it either gives 0,or 1. We might have a situation where if something is very near T we want to be a bit uncertain if it is cold or hot)

Logistic Regression uses the sigmoid function, that is a bit like a "cut-off" function (a function that is 0 up to a point, then at some X_0 very quickly goes to 1). This allows us to split values of x to two groups-classes: those less than X_0, and those more than X_0. At the same time, it is a bit better than a step function ) because sigmoid is smooth and so can have a derivative. (Also, maybe if we are very close to the cut-off point we might not want to give 1 or 0 but rather a number)

Ideally you want a function that would give you a good approximation of the probability some sample belongs to class 0 or 1 given its value, and the sigmoid function can help you there too, but the first paragraph can help for intuition.