I am trying to understand the math behind logistic regression. Going through a couple of websites, lectures and books, I tried to derive the cost function by thinking of it as the negative of the maximum likelihood. My derivation matches the cost function shown in this Wikipedia page https://en.wikipedia.org/wiki/Logistic_regression and in other places.

If the inputs are $x^{(i)}$ and outputs are $y^{(i)}$, where $(i)$ refers to the $i$th data point, then the cost as a function of weights $w$ seems to be

$$-\sum_{i=1}^m y^{(i)} \log\left(\frac{1}{1 + e^{-w^Tx^{(i)}}}\right)+\left(1-y^{(i)} \right) \log \left( 1-\frac{1}{1 + e^{-w^Tx^{(i)}}} \right) $$

I can simplify further to $$\sum_{i=1}^m -w^Tx^{(i)} y^{(i)} +\log⁡ (1+e^{-w^Tx^{(i)}})$$ However the expression shown in the SciKit Learn guide https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression is $$\sum_{i=1}^m \log ( 1 + e^{-w^Tx^{(i)}y^{(i)}}) $$

I have tried some algebra and am not able to derive their formulation. Am I missing something? It is highly possible that I haven't tried all the tricks there are in simplifying

  • $\begingroup$ Important note: scikit-learn's implementation of logistic regression uses regularization by default. $\endgroup$
    – littleO
    Feb 21 '21 at 7:51

From the same scikit-learn link, note that, in this notation, it’s assumed that the target $y^{(i)}$ takes values $\{-1,+1\}$ in the set at trial $i$.

To the contrary, $y^{(i)} \in \{0,1\}$ as per your initial definition of the binary cross entropy (BCE) cost function.

In the scikit-learn notation, we have

$$P(y^{(i)}=+1\mid x^{(i)})=\sigma(w^Tx^{(i)})=\frac{1}{1+e^{-w^Tx^{(i)}}}$$

$$P(y^{(i)}=-1\mid x^{(i)})=1-\sigma(w^Tx^{(i)})=\frac{1}{1+e^{w^Tx^{(i)}}},$$

so that in both cases we have

$$P(y^{(i)}\mid x^{(i)})=\frac{1}{1+e^{-w^Tx^{(i)}y^{(i)}}}$$

With independence assumption, the likelihood is $\prod\limits_{i=1}^m \frac{1}{1+e^{-w^Tx^{(i)}y^{(i)}}}$

The negative log-likelihood is $\sum\limits_{i=1}^m \log(1+e^{-w^T x^{(i)} y^{(i)}})$, which is the cost function to be minimized (for MLE), along with some ($L_2$) regularization term.

  • $\begingroup$ Nice! Thanks. I don't think my original derivation (not really mine, courtesy Wikipedia) makes any assumptions about the actual values for y. This is good to know about sci-kit learn $\endgroup$
    – Anu
    Feb 20 '21 at 22:31
  • 1
    $\begingroup$ Never mind, I see where the assumption of y in {0,1} is made in the derivation. $\endgroup$
    – Anu
    Feb 20 '21 at 22:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.