Based on Andrew Ng's Coursera machine learning course, logistic regression has the following cost function (probably among others): $$ \operatorname{cost}(h(x),y) = \begin{cases} -\log(h(x)), & \text{if $y$ = 1} \\ -\log(1-h(x)), & \text{if $y$ = 0} \end{cases} $$ where $y$ is either $0$ or $1$ and $h(x)$ is a sigmoid function returning inclusively between $[0, 1]$.

According to the class, this can be simplified to the following form: $$\operatorname{cost}(h(x),y)=-y \cdot \log(h(x))-(1-y) \cdot \log(1-h(x))$$

When both $y$ and $h(x)$ are $1$ then the second part of the formula becomes $(0) \cdot log(0)$ which is $0 \cdot (-\infty)$ which is indeterminate.

My question is, providing what I wrote above is correct, how this can still work? I see numerous working Matlab/Octave implementations which don't handle differently this edge case.

  • $\begingroup$ The same thing occurs if $y$ and $h(x)$ are both 0. The "simplification" is not equivalent to the piecewise cost function. Such simplifications are convenient (one line, avoids branching), but they often have singularities (real or due to floating point math) as you point out. These may or may not be a problem in actual use. If you want an implementation that handles this case, then you'll need to modify the code a bit. Also, true sigmoids only approach 0 and 1 asymptotically though your implementation may return 0 and 1 for +/- large values or Inf. $\endgroup$ – horchler Mar 17 '16 at 17:44
  • $\begingroup$ @horchler, thank you for your comment! If you copied it into an answer, I'd gladly accept it :). $\endgroup$ – szotsaki Mar 18 '16 at 10:19

The only way to get $h(x)$ to be $0$ is when $x = -\infty$. But inf is region not a value no matter which value of x you take there's always a value greater than it(or smaller than it in case of -inf). So the h(x) value never equals 0. The catch here is as the x tends to get closer to -inf the value of h(x) gets closer to 0, but not the exact value of 0.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.