10
$\begingroup$

How would you show that if you do logistic regression with a squared loss function, it is not a convex optimization problem (in parameters)?

In other words, your loss function for an individual observation is $(y - p)^2$, where $y$ is the dependent variable and $p$ is your prediction, and $$p = \frac{1}{1+\exp(-w^Tx)},$$ where $x$ is the vector of predictors and $w$ is the vector of weights.

$\endgroup$
  • $\begingroup$ You could show that the definition of a convex function is violated. It just takes one $w$, one $y$, two $x$ vectors, and a $\lambda$. $\endgroup$ – LinAlg Oct 25 '16 at 21:34
7
$\begingroup$

As long as $x$ is not zero, the squared error loss with respect to $w$ will be non-convex.

Here is a graph that shows both the squared error loss and the log loss of the sigmoid function: https://www.desmos.com/calculator/kxz6lzszf9

You can see that the squared error loss (red and orange curves) is non-convex, whereas the log loss (green and blue curves) is convex.


To follow up with Qiang Chen's answer, the red and orange curves are the squared loss functions in question, and they are both non-convex (not fully convex). They are also non-concave (not fully concave). They are half convex on one side and half concave on the other, with a middle inflection point where $f(x)=0$. Convex and concave functions are ones that always curve in the same direction, either up or down, they can not have that kind of "S" shape that both the red and orange curves have.

All the $x_1$ and $x_2$ values on the concave side of the function do not satisfy that definition of a convex function.

$\endgroup$
-1
$\begingroup$

I think it's convex, from Elliot Waite's demo: https://www.desmos.com/calculator/kxz6lzszf9, you can not find any data avoid the definition of convex function

picture formula of definition

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.