Why Logistic Regression for Classification Problems?

In a class on machine learning, we covered classification problems. In such a problem, you are studying a property of some object, say malignity of tumors in a patient. You are first given a training set, which consists of a set of ordered pairs $(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), ... (x^{(n)}, y^{(n)})$, where each $x^{(i)}$ is a vector of desired parameters (in the tumor example, size could be such a parameter), while each $y^{(i)}$ is either zero or one depending on the status of this property (in the tumor example, one might say a tumor is benign if $y^{(i)} = 0$, and malign if $y^{(i)}=1$. By fitting a function to this training set, one finds a hypothesis function, which hopefully predicts whether tumors are benign or malign in the future. But how to fit such a function? The professor of the class gave an example that wouldn't work, linear regression. Linear regression was shown as a poor method because outliers in the training set would influence the hypothesis too drastically. Then the professor said that the better method for classification problems was logistic regression. However, he did not explain this -- from an observer's point of view, it seemed logistic regression was chosen ad hoc as a fitting method. Could someone please ex

-

Maybe your professor should ex, since he's the one who introduced the logistic regression? One (possibly weak) reason to use the logistic curve is: it is one of the simplest functions one can think of that increases from 0 to 1, always staying between 0 and 1. Thus, its value at $x$ can be interpreted as the likelihood of $x$ belonging to the "$y=1$" group.

-

Logistic Regression uses the sigmoid function, that is a bit like a "cut-off" function (a function that is 0 up to a point, then at some X_0 very quickly goes to 1). This allows us to split values of x to two groups-classes: those less than X_0, and those more than X_0. At the same time, it is a bit better than a step function (e.g. the function $$f(x)= \{\begin{array}{l} 0,x<X_0 \\ 1,x\geq X_0\end{array}$$ ) because sigmoid is smooth and so can have a derivative. (Also, maybe if we are very close to the cut-off point we might not want to give 1 or 0 but rather a number)

Ideally you want a function that would give you a good approximation of the probability some sample belongs to class 0 or 1 given its value, and the sigmoid function can help you there too, but the first paragraph can help for intuition.

-