Neural network - function estimation My questions are about the paper: Semi-supervised Learning with Deep Generative Models (Kingma, D.P. et al, 2014).
Suppose I have a generative network with input $x$, and a hidden layer with hidden nodes $z$. We use $z$ to generate $x$.
Now suppose my prior distribution on the hidden units is a spherical Gaussian:
$p(z) \sim N(z \mid 0, I)$
And I want to obtain a posterior distribution $q_\phi (z \mid x) = N(z \mid \mu_\phi(x), {\rm{diag}}(\sigma_{\phi}^2 (x)))$.
I parameterize a multi-layer perceptron (MLP) and use that MLP to learn $\mu_{\phi}(x)$ and $\sigma_{\phi}(x)$.
My questions are:
1) How do the parameters $\phi$ of the MLP determine the mean and variance? 
2) How does that work mathematically? How am I getting the posterior mean and variance from this MLP?
3) Why do I need an MLP at all?
 A: (1) I'd think of it like this. Let $f_\phi(x)$ be the neural network with weights $\phi$, such that $(\mu_\phi(x), \sigma_\phi(x))=f_\phi(x)$. In other words, if 
$\mu,\sigma\in\mathbb{R}^d$ and $x\in\mathbb{R}^n$, then we are learning a map $f_\phi:\mathbb{R}^n\rightarrow\mathbb{R}^{2d}$, assuming a diagonal covariance. So, yes, the output of the neural network is simply the parameters of the posterior Gaussian. The encoder parameters $\phi$ determine $\mu$ and $\sigma$ distributional parameters indirectly, via the operations of the neural network.
(2) Mathematically, it can be confusing why $q_\phi$ is Gaussian. Indeed, the true posterior: $$ p(z|x)=\frac{p(x|z)p(z)}{p(x)} $$ is usually not Gaussian. In fact, the whole reason we are doing this is that $p(z|x)$ is actually not tractable because $p(x)$ is generally very difficult to estimate; hence, we are simply approximating it with $q$, i.e. $q_\phi(z|x)\approx p(z|x)$. Variational techniques allow us to tractably optimize $\phi$ such that this approximation is decent. As for how the MLP gives us $\mu$ and $\sigma$, see (1) just above.
(3) As for why an MLP is needed, it is because the problem of approximating $p(z|x)$ with $q_\phi$ is a function approximation problem, which is precisely what MLPs are good for. Technically, you could use any other statistical learning technique to estimate $\mu$ and $\sigma$; it just so happens that MLPs are very powerful and efficient, i.e. capable of tractably representing arbitrarily functions. Just sit back and ask yourself: how can I approximate a conditional distribution over $z$, given the high-dimensional nature of $x$? Essentially, you need a function that is given a data point $x$ and returns a probability density over the latent space $z$. MLPs are simply good tools for achieving this.

Useful papers for introductory VAEs:


*

*Auto-Encoding Variational Bayes by Kingma & Welling

*Stochastic Backpropagation and Approximate Inference in Deep Generative Models by Rezende et al
