# Neural network - function estimation

My questions are about the paper: Semi-supervised Learning with Deep Generative Models (Kingma, D.P. et al, 2014).

Suppose I have a generative network with input $x$, and a hidden layer with hidden nodes $z$. We use $z$ to generate $x$.

Now suppose my prior distribution on the hidden units is a spherical Gaussian:

$p(z) \sim N(z \mid 0, I)$

And I want to obtain a posterior distribution $q_\phi (z \mid x) = N(z \mid \mu_\phi(x), {\rm{diag}}(\sigma_{\phi}^2 (x)))$.

I parameterize a multi-layer perceptron (MLP) and use that MLP to learn $\mu_{\phi}(x)$ and $\sigma_{\phi}(x)$.

My questions are:

1) How do the parameters $\phi$ of the MLP determine the mean and variance?

2) How does that work mathematically? How am I getting the posterior mean and variance from this MLP?

3) Why do I need an MLP at all?

• sorry I don't get it. You write $z| x$ so you use $x$ to generate $z$ : in simple words, $Z$ is Gaussian a random variable whose mean and variance depend on some parameters $x$ and $\theta$. Jun 30, 2016 at 20:37
• And you want to use a neural network for learning $\theta$ knowing : a (neural network) model for $\mu,\sigma$ in term of $x,\theta$, and some examples $x_i,z_i$. I.e. you will maximize over $\theta$ the log-likelihood $\sum_i \log p(z_i | x_i,\theta) = \sum_i \log p(z_i) + \log \mathcal{N}(z_i | \mu_\theta(x_i,\theta),\sigma_\theta(x_i,)^2)$ where $\mu_\theta(x),\sigma_\theta(x)^2$ are neural network functions of the input $x$ with weights $\theta$ Jun 30, 2016 at 20:44
• overall, if $x$ are vectors of $\mathbb{R}^3$, you will train one neural network with $k$ inputs neurons (for $x$) and $2$ output neurons ($\mu$ and $\sigma$) Jun 30, 2016 at 20:47
• Yep, sorry, that was confusing! This is a generative network. So values from z are sampled from the prior for z, and then x is generated according to p(x | z). The distribution q is the posterior distribution for z, given set values of x. Jun 30, 2016 at 22:58
• where $x_i,z_i$ come from doesn't change anything... Jun 30, 2016 at 23:03

(1) I'd think of it like this. Let $f_\phi(x)$ be the neural network with weights $\phi$, such that $(\mu_\phi(x), \sigma_\phi(x))=f_\phi(x)$. In other words, if $\mu,\sigma\in\mathbb{R}^d$ and $x\in\mathbb{R}^n$, then we are learning a map $f_\phi:\mathbb{R}^n\rightarrow\mathbb{R}^{2d}$, assuming a diagonal covariance. So, yes, the output of the neural network is simply the parameters of the posterior Gaussian. The encoder parameters $\phi$ determine $\mu$ and $\sigma$ distributional parameters indirectly, via the operations of the neural network.

(2) Mathematically, it can be confusing why $q_\phi$ is Gaussian. Indeed, the true posterior: $$p(z|x)=\frac{p(x|z)p(z)}{p(x)}$$ is usually not Gaussian. In fact, the whole reason we are doing this is that $p(z|x)$ is actually not tractable because $p(x)$ is generally very difficult to estimate; hence, we are simply approximating it with $q$, i.e. $q_\phi(z|x)\approx p(z|x)$. Variational techniques allow us to tractably optimize $\phi$ such that this approximation is decent. As for how the MLP gives us $\mu$ and $\sigma$, see (1) just above.

(3) As for why an MLP is needed, it is because the problem of approximating $p(z|x)$ with $q_\phi$ is a function approximation problem, which is precisely what MLPs are good for. Technically, you could use any other statistical learning technique to estimate $\mu$ and $\sigma$; it just so happens that MLPs are very powerful and efficient, i.e. capable of tractably representing arbitrarily functions. Just sit back and ask yourself: how can I approximate a conditional distribution over $z$, given the high-dimensional nature of $x$? Essentially, you need a function that is given a data point $x$ and returns a probability density over the latent space $z$. MLPs are simply good tools for achieving this.

Useful papers for introductory VAEs:

• Thank you! Great explanation. So, in part (1), we're using a function of the weights (an MLP) to map from our input space (x in R^n) to a 2d-dimensional latent space (mu, sigma in R^2d). So is it that we start by assuming diagonal covariance and then learn the true covariance? Jul 13, 2018 at 15:12
• @StatsSorceress Essentially yes, although I'd say you are mapping from the input space to the $2d$ space of parameters of Gaussians over the latent space. In other words, mapping to the space of $(\mu,\sigma)$, not directly to $z$ (which is only $d$-dimensional). Given the parameters, you can then sample a $z$ value (so it's a stochastic map from $x$ to $z$). Jul 13, 2018 at 20:21
• For the covariance, the diagonal assumption is restrictive. It harms the accuracy of the conditional distribution, but it allows fast and efficient variational optimization. You do not necessarily need to assume diagonality (nor Gaussianity even). It just adds computation time for a better representation of the conditional (but for generative models we might not care that much because we might just want the decoder rather than the encoder for our downstream task). (Incidentally I'm glad to see it was helpful even after two years lol; also, great username). Jul 13, 2018 at 20:24
• Wow! No one ever told me I was mapping to a space of parameters! I thought I was mapping to z, the latent space. Okay, that changes my entire perspective. Thank you so much for taking the time. Jul 14, 2018 at 20:38