The definition of the logistic regression loss function I use is this:

enter image description here

We draw the data i.i.d. according to some distribution $D$, realised by some $\langle X, Y \rangle$. Now if $h_w$ was the real data generating distribution, that is $\mathbb{P}(Y = 1 \ | \ X = x) = h_w(x)$, I'd expect the loss function to be zero, but I do not think this is the case. Is that a calculation mistake or is it unreasonable to ask the loss function to be zero for a perfect hypothesis? If so, why?


$$\mathbb{E}[\log(1 + exp(-Y\langle w, X \rangle))] = \mathbb{E}[\mathbb{E}[\log(1 + exp(-Y\langle w, X \rangle))] \ | \ X = x]$$

$$= \int \log(1 + exp(-\langle w, x \rangle)) \cdot \frac{1}{1 + e^{-\langle w, x \rangle}} + \log(1 + exp(\langle w, x \rangle)) \cdot \frac{1}{1 + e^{\langle w, x \rangle}} d\mathbb{P}^X(x)$$

and this is not zero because the function we integrate is not zero almost everywhere.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.