How to understand Variational Autoencoder As new to variational autoencoder, there are some simple details perplex me. The basic idea of VAE is to use an encoder to map some unknown distribution (e.g. mnist images) to a specific distribution like Gaussian, and then decode this latent distribution back to the original distribution. In Kingma's paper, I feel confused about these simple facts: 


*

*The paper assume p(x|z) to be a Gaussian distribution. But in my opinion, p(x|z) is what the decoder does to map the latent variable z to the original distribution(e.g. mnist images). Thus, p(x|z) actually represents the original distribution. Why can we just simply assume it to be a Gaussian distribution?

*The cost function of VAE is composed of the reconstruction error and regularization error. But I don't know what's the effect of regularization error and what if we just use the reconstruction error?

*The encoder of VAE outputs the mean u and s.t.d for the Gaussian distribution and then we sample z from it. But why don't we just let the encoder output z in Gaussian just as what we do in GAN(In GAN, we use a generator to directly output variable in the distribution we want)? 

 A: Below are my answers to your questions. An excellent explanation of variational autoencoder can be found at in this blog.


*

*Variational autoencoder assumes the marginal likelihood of data as
$$p(x) = \int dz p(x|z)p(z)$$
i.e. a continuous mixture of Gaussian. This mixture is a universal approximator for any distributions. 

*The effect of regularization error is to make the posterior distribution of latent variable $p(z|x)$ look like a unit Gaussian. The reason we need a regularization error (aside from the fact that it comes naturally from the derivation) is we want to limit the range of latent variables so that similar data lead to similar latent variables. For example, suppose we have many different images of number "9", without the regularization error nothing can prevent the network from mapping these images to completely different value far away in the latent space, while the regularization error will make it more favorable to map these images to values close in the latent space. So you can see its similarity with other regularization techniques in neural network in here.

*The random sample for GAN is drawn from the prior distribution of latent variable $p(z)$, and the random sample for variational autoencoder is drawn from the posterior distribution of latent variable $p(z|x)$. If you simply want to generate an image $x$, then you do not need $p(z|x)$. You can just draw from 
$p(z)$ as you did in GAN and pass $z$ through the decoder network.
