What is the connection of a sigmoid with probability? In high school probabilities for an event are the chance of an event to happen. The easiest is a coin with 1/2 for each event.
I've been reading about Machine Learning and sigmoid functions, and it seems sigmoid has something to do with probability.
I've noticed the derivative is a Bell-like curve, and sure enough the integral of bell curves is normally a sigmoid.
But apart from being between 0 and 1 I see no real connection to a probability.
Any hints or useful places to get started?
 A: The sigmoid function $\sigma(u) = \frac{e^u}{1 + e^u}$ is useful in machine learning because it converts a real number into a probability (that is, just a number between $0$ and $1$). If you try to come up with the simplest possible differentiable function that does this job for us, I don't think you can find anything much simpler or more elegant than the sigmoid function. One of the elegant properties of the sigmoid function is that $\sigma'(u) = \sigma(u)(1 - \sigma(u))$.
When doing binary classification (with two classes called class $0$ and class $1$), we want to find a prediction function $f$ that takes a feature vector $x$ as input and returns as output a probability that an example described by this feature vector belongs to class $1$. We use the sigmoid function to force the output of $f$ to be between $0$ and $1$ so that it can be interpreted as a probability.

By the way, another elegant property of the sigmoid function is that it "plays well" with the cross-entropy loss function $\ell(p,q) = -p\log(q) - (1 - p) \log(1 - q)$, which is used in machine learning to measure how well a probability $q$ agrees with another probability $p$. Notice that if $0 \leq p \leq 1$ then the function $L(u) = \ell(p, \sigma(u))$ satisfies
\begin{align}
L(u) &= \ell(p, \sigma(u)) \\
&= -p \log\left(\frac{e^u}{1 + e^u} \right) - (1 - p) \log\left( 1 - \frac{e^u}{1 + e^u} \right) \\
&= -p \log\left(\frac{e^u}{1 + e^u} \right) + (1-p) \log\left(1 + e^u \right) \\
&= -pu + \log(1 + e^u) .
\end{align}
Look how much that simplified! Beautiful.
It follows that
\begin{align}
L'(u) &= \sigma(u) - p.
\end{align}
That's a beautiful and simple result. When using gradient descent to train a logistic regression model, the iteration is very simple and intuitive thanks to the simplicity of this result.
