If we are doing a binary classification using logistic regression, we often use the cross entropy function as our loss function. More specifically, suppose we have $T$ training examples of the form $(x^{(t)},y^{(t)})$, where $x^{(t)}\in\mathbb{R}^{n+1},y^{(t)}\in\{0,1\}$, we use the following loss function $$\mathcal{LF}(\theta)=-\dfrac{1}{T}\sum_{t}y^{t}\log(\text{sigm}(\theta^T x))+(1-y^{(t)})\log(1-\text{sigm}(\theta^T x))$$ , where $\text{sigm}$ denotes the sigmoid function.
However, if we are doing linear regression, we often use squared-error as our loss function. Are there any specific reasons for using the cross entropy function instead of using squared-error or the classification error in logistic regression? I read somewhere that, if we use squared-error for binary classification, the resulting loss function would be non-convex. Is it the only reason reason, or is there any other deeper reason which I am missing?
To get a sense of how different loss functions would look like, I have generated $50$ random datapoints on both sides of the line $y=x$. I have assigned the class $c=1$ to the datapoints which are present on one side of the line $y=x$, and $c=0$ to the other datapoints. After generating this data, I have computed the costs for different lines $\theta_1 x-\theta_2y=0$ which passes through the origin using the following loss functions:
- squared-error function using the predicted labels and the actual labels.
- squared-error function using the continuous scores $\theta^Tx$ instead of thresholding by $0$.
- squared-error function using the continuous scores $\text{sigm}(\theta^T x)$.
- classification error, i.e., number of misclassified points.
- cross entropy loss function.
I have considered only the lines which pass through the origin instead of general lines, such as $\theta_1x-\theta_2y+\theta_0=0$, so that I can plot the loss function. I have obtained the following plots.
From the above plots, we can infer the following:
- The plot corresponding to $1$ is neither smooth, it is not even continuous, nor convex. This makes sense since the cost can take only finite number of values for any $\theta_1,\theta_2$.
- The plot corresponding to $2$ is smooth as well as convex.
- The plot corresponding to $3$ is smooth but is not convex.
- The plot corresponding to $4$ is neither smooth nor convex, similar to $1$.
- The plot corresponding to $5$ is smooth as well as convex, similar to $2$.
If I am not mistaken, for the purpose of minimizing the loss function, the loss functions corresponding to $(2)$ and $(5)$ are equally good since they both are smooth and convex functions. Is there any reason to use $(5)$ rather than $(2)$? Also, apart from the smoothness or convexity, are there any reasons for preferring cross entropy loss function instead of squared-error?