1
$\begingroup$

I'm currently learning machine learning logistic regression and I'm really confused with logistic loss. I know the formula for logistic loss is $\displaystyle g(z)=\frac{1}{1+e^{-z}}$ here $z = W^TX_i$

But I've read that this is a non convex function. So we usually take logarithm of logistic loss to make it a convex function for optimization techniques like gradient descent etc..

$$J(\theta) =-\frac{1}{m}\sum_{i=1}^{m}y^{i}\log(h_\theta(x^{i}))+(1-y^{i})\log(1-h_\theta(x^{i}))$$ $$h_\theta(x^i) = \frac{1}{1+e^{-z}}$$

How can we know if a functions/loss is convex or non convex? Why are we taking log for logistic loss? How can that makes the loss function convex?

$\endgroup$
2
$\begingroup$

$g$ is not convex: consider the points $0,1,N$ where $N$ is a positive integer $>1$. We have $1=\frac 1 N (N)+(1-\frac 1 N) 0$. If $g$ is convex then we would have $g(1) \leq \frac 1 N g(N)+(1-\frac 1 N)g(0)$. If you let $N \to \infty$ in this you get the contradiction that $e \leq 1$.

$\log g$ is also not convex. It is concave. So $-\log\, g$ is convex. To see this write $-\log\, g$ as $-\log \, \frac {e^{x}} {1+e^{x}}=-x +\log \, (1+e^{x})$. it is easy to calculate the second derivative of the function and show that it is positive.

$\endgroup$
  • $\begingroup$ So if the second derivative of a function is positive then it's a convex function right? Can you simplify it a bit more? $\endgroup$ – user_6396 Jul 8 at 23:59
  • 1
    $\begingroup$ Yes, a twice differentiable function is convex iff its second derivative is non-negative iff its first derivative is monotonically increasing. $\endgroup$ – Kabo Murphy Jul 9 at 0:00
  • $\begingroup$ Thank you for the help. So how can I know the first derivative is monotonically increasing in the case of negative log(logistic loss). Can you derive it? An example? I'm trying to get to understand it better. $\endgroup$ – user_6396 Jul 9 at 0:37
  • 1
    $\begingroup$ The first derivative is $-1+\frac {e^{x}} {1+e^{x}}=-\frac 1 {1+e^{x}}$. Since the denominator is increasing the ratio is decreasing and the minus sign makes it increasing. $\endgroup$ – Kabo Murphy Jul 9 at 0:43
  • 1
    $\begingroup$ No. To show that $g$ is not convex I did not use derivatives at all. I used just the definition of a convex function. But to show that $-\log\, g$ is convex I used derivatives. So the derivative I calculated are for $-\log\, g$. $\endgroup$ – Kabo Murphy Jul 9 at 5:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.