2
$\begingroup$

I'm currently learning machine learning logistic regression and I'm really confused with logistic loss. I know the formula for logistic loss is $\displaystyle g(z)=\frac{1}{1+e^{-z}}$ here $z = W^TX_i$

But I've read that this is a non convex function. So we usually take logarithm of logistic loss to make it a convex function for optimization techniques like gradient descent etc..

$$J(\theta) =-\frac{1}{m}\sum_{i=1}^{m}y^{i}\log(h_\theta(x^{i}))+(1-y^{i})\log(1-h_\theta(x^{i}))$$ $$h_\theta(x^i) = \frac{1}{1+e^{-z}}$$

How can we know if a functions/loss is convex or non convex? Why are we taking log for logistic loss? How can that makes the loss function convex?

$\endgroup$
1
  • $\begingroup$ "A differentiable function of one variable is convex on an interval if and only if its derivative is monotonically non-decreasing on that interval. If a function is differentiable and convex then it is also continuously differentiable"--en.wikipedia.org/wiki/Convex_function#Functions_of_one_variable $\endgroup$
    – saulspatz
    Jul 8, 2019 at 23:49

1 Answer 1

2
$\begingroup$

$g$ is not convex: consider the points $0,1,N$ where $N$ is a positive integer $>1$. We have $1=\frac 1 N (N)+(1-\frac 1 N) 0$. If $g$ is convex then we would have $g(1) \leq \frac 1 N g(N)+(1-\frac 1 N)g(0)$. If you let $N \to \infty$ in this you get the contradiction that $e \leq 1$.

$\log g$ is also not convex. It is concave. So $-\log\, g$ is convex. To see this write $-\log\, g$ as $-\log \, \frac {e^{x}} {1+e^{x}}=-x +\log \, (1+e^{x})$. it is easy to calculate the second derivative of the function and show that it is positive.

$\endgroup$
9
  • $\begingroup$ So if the second derivative of a function is positive then it's a convex function right? Can you simplify it a bit more? $\endgroup$
    – Jeeth
    Jul 8, 2019 at 23:59
  • 1
    $\begingroup$ Yes, a twice differentiable function is convex iff its second derivative is non-negative iff its first derivative is monotonically increasing. $\endgroup$ Jul 9, 2019 at 0:00
  • $\begingroup$ Thank you for the help. So how can I know the first derivative is monotonically increasing in the case of negative log(logistic loss). Can you derive it? An example? I'm trying to get to understand it better. $\endgroup$
    – Jeeth
    Jul 9, 2019 at 0:37
  • 1
    $\begingroup$ The first derivative is $-1+\frac {e^{x}} {1+e^{x}}=-\frac 1 {1+e^{x}}$. Since the denominator is increasing the ratio is decreasing and the minus sign makes it increasing. $\endgroup$ Jul 9, 2019 at 0:43
  • 1
    $\begingroup$ No. To show that $g$ is not convex I did not use derivatives at all. I used just the definition of a convex function. But to show that $-\log\, g$ is convex I used derivatives. So the derivative I calculated are for $-\log\, g$. $\endgroup$ Jul 9, 2019 at 5:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .