0
$\begingroup$

The definition of the logistic regression loss function I use is this:

enter image description here

We draw the data i.i.d. according to some distribution $D$, realised by some $\langle X, Y \rangle$. Now if $h_w$ was the real data generating distribution, that is $\mathbb{P}(Y = 1 \ | \ X = x) = h_w(x)$, I'd expect the loss function to be zero, but I do not think this is the case. Is that a calculation mistake or is it unreasonable to ask the loss function to be zero for a perfect hypothesis? If so, why?

Calculation

$$\mathbb{E}[\log(1 + exp(-Y\langle w, X \rangle))] = \mathbb{E}[\mathbb{E}[\log(1 + exp(-Y\langle w, X \rangle))] \ | \ X = x]$$

$$= \int \log(1 + exp(-\langle w, x \rangle)) \cdot \frac{1}{1 + e^{-\langle w, x \rangle}} + \log(1 + exp(\langle w, x \rangle)) \cdot \frac{1}{1 + e^{\langle w, x \rangle}} d\mathbb{P}^X(x)$$

and this is not zero because the function we integrate is not zero almost everywhere.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.