Arriving at the Logistic function from a Binomial Distribution and Maximum Likelihood I've been trying to understand the origin of the Logistic function in Logistic regression:
$$\Pr(Y=1|x;\theta)=\frac{1}{1+e^{-\theta x}}$$
I was lead to beilive that one could somehow arrive at this purely from the Binomial Distribution and a Maximum Likelihood type argument, but I can't quite see it. Is seems that if one considers the Binomial Distribution as a member of the Exponential Family of distributions the Logit function arises as the "natural parameter", but I'm not quite sure as to the meaning or consequences of this.
To summarize:


*

*Is the logistic function "optimal" in some mathematical way, or is
this just a convenient function? If it is, can one derive it from a Maximal Likelihood formulation? 

*How (if at all) is all this connected to the Exponential Family?


I've searched around quite a lot, as well as tried to derive this myself, but so far no dice. 
Any ideas?
 A: Writing the probability mass function of a bernoulli random variable $X$ with
parameter $\pi$ (let's simplify things here) and then introducing the
exponential function
\begin{eqnarray*}
  p \left( x ; \pi \right) & = & \pi^x \left( 1 - \pi \right)^{1 - x}\\
  & = & \exp \left( x \log \left( \pi \right) + \left( 1 - x \right) \log
  \left( 1 - \pi \right) \right)\\
  & = & \exp \left( x \log \left( \frac{\pi}{1 - \pi} \right) + \log \left( 1
  - \pi \right) \right)
\end{eqnarray*}
From the above, we see that this is in the form of an exponential family with
statistic $x$ and parameter $\log \left( \frac{\pi}{1 - \pi} \right)$ (the
remaining $\log \left( 1 - \pi \right)$ is just a constant of integration).
It is a natural to write this in canonical form by making the transformation
$\theta = \log \left( \frac{\pi}{1 - \pi} \right)$ in order to get something
of the form $p \left( x ; \theta \right) = \exp \left( x \theta - c \left(
\theta \right) \right)$. This new function is the logit function. If you would
like to express the inverse relationship function you obtain the logistic
transformation
$$ \pi = \frac{1}{1 + \exp \left( - \theta \right)} $$
Regarding you two questions, and as far as I understand the issues: The
logistic function rises from the Bernoulli distribution. That is as natural as
you can can get. In that framework, the linear form $x \theta$ has the natural
decomposition sufficient statistic vs. parameter. That is, taking the distribution of $n$ copies of $X$, the mean is a
sufficient statistic for estimating $\theta$ and you do not need to go beyond
that for the maximum likelihood estimator. I think this is the link you were
looking for.
A: Let $Y \sim Binomial(n, \phi),$ where $n$ denotes a fixed number of trials and $\phi$ denotes the probability of success. We want to show that $Y \sim ExponentialFamily(\eta).$
To do so, we rewrite the probability mass function $p(y) = \binom{n}{y} \phi^{y} (1 - \phi)^{n-y}$ such that $p(y) = b(y) \exp(\eta y - a(\eta))$ for some base measure $b(y),$ natural parameter $\eta,$ sufficient statistic $T(y)=y,$ and log-partition function $a(\eta).$ Namely, we have that
$$p(y) = \binom{n}{y} \phi^{y} (1 - \phi)^{n-y} = \binom{n}{y} \exp(\ln(\frac{\phi}{1-\phi})y - n\ln(\frac{1}{1-\phi})),$$
which implies that $b(y) = \binom{n}{y},$ $\eta = \ln(\frac{\phi}{1-\phi}),$ $T(y)=y,$ and $a(\eta) = n\ln(\frac{1}{1-\phi}).$
Now, focus on the natural parameter, which is equal to the log odds, i.e., we see a map from the probability of success to the natural parameter. In turn, we ask what is the inverse map from the natural parameter to the probability of success? To find it, we begin with
$$e^{\eta} = \frac{\phi}{1 - \phi},$$ then we see that
$$\phi = \frac{e^{\eta}}{e^{\eta} + 1} = \frac{1}{1 + e^{-\eta}}.$$ Hence, the inverse map is the logistic function.
Lastly, if we assume $Y \sim Bernoulli(\phi),$ then we arrive at the same conclusion; and this is what is shown by Learner's solution.
A: Another way to see how the logistic function arises is to consider $Y \sim Bernoulli(\phi),$ as well as the value of another random variable $X=x.$ Then, rewrite the following conditional probability in terms of Bayes' rule
$$P(Y=1 | X=x) = \frac{P(Y=1)f_{1}(x)}{P(Y=1)f_{1}(x) + P(Y=0)f_{0}(x)} = \frac{\phi f_{1}(x)}{\phi f_{1}(x) + (1 - \phi)f_{0}(x)},$$
where $\phi$ denotes the probability of success and $f_{i}(x)$ denotes the conditional probability mass or density function for $i=0,1.$
Now, suppose $\phi \neq 0 \neq f_{1}(x)$ in order to obtain
$$\frac{1}{1 + \frac{1 - \phi}{\phi}\frac{f_{0}(x)}{f_{1}(x)}}.$$ Then, rewrite the quantity such that it is equal to
$$\frac{1}{1 + \exp({-(\ln(\frac{\phi}{1 - \phi}) + \ln(\frac{f_{1}(x)}{f_{0}(x)}))})} = \frac{1}{1 + \exp(-z)},$$
where we recognize the right-hand side as the logistic function.
