# Reparameterization Trick for Gaussian Distribution

What is the true equation for the reparameterization trick for gaussian distribution ? I see many forms

$z = \mu + \sigma \times \epsilon$ http://kvfrans.com/variational-autoencoders-explained/

$z = \mu + \sqrt{ \exp(\sigma) } \times \epsilon$ https://jmetzen.github.io/2015-11-27/vae.html

$z = \mu + \exp(\frac{\sigma}{2}) \times \epsilon$ https://github.com/fchollet/keras/blob/master/examples/variational_autoencoder.py

How do I derive the one true equation ? Or can I do it anyway I want as long as I add mu and a function of sigma ?

• hi anyone help thanks
– Kong
Nov 28, 2017 at 10:55
• hi anyone help thanks
– Kong
Nov 28, 2017 at 20:14
• hi anyone help thanks
– Kong
Nov 29, 2017 at 9:39
• hi anyone help thanks free 50 points
– Kong
Dec 3, 2017 at 15:06
• Oh my word. You need to add more context. Your question, along with the links, do not help substantiate the question you are asking. Dec 4, 2017 at 3:15

There are a few different quantities to distinguish here:

• $\sigma,$ the standard deviation
• $\sigma^2,$ the variance
• $\log(\sigma^2)$

These can all be easily converted to one another - they are just different ways of measuring the same thing. In VAE, the encoder will be trained to generate one of these parameters, and you need to interpret the output accordingly. If the encoder was trained to produce $\sigma,$ you should (presumably) use its output as $\sigma.$ If the encoder was trained to produce $\log(\sigma^2),$ you should use its output as $\log(\sigma^2);$ apply the transformation $x\mapsto \exp(x/2)$ to get $\sigma.$

To generate a normally distributed sample $X$ mean mean $\mu$ and standard deviation $\sigma$ given a standard normal $Z\sim N(0,1),$ use the expression

$$X=\mu+\sigma Z.$$

Assume we have a normal distribution $q$ that is parameterized by $\theta$, specifically $q_{\theta}(x) = N(\theta,1)$. We want to solve the below problem $$\text{min}_{\theta} \quad E_q[x^2]$$ We want to understand how the reparameterization trick helps in calculating the gradient of this objective $E_q[x^2]$.

One way to calculate $\nabla_{\theta} E_q[x^2]$ is as follows $$\nabla_{\theta} E_q[x^2] = \nabla_{\theta} \int q_{\theta}(x) x^2 dx = \int x^2 \nabla_{\theta} q_{\theta}(x) \frac{q_{\theta}(x)}{q_{\theta}(x)} dx = \int q_{\theta}(x) \nabla_{\theta} \log q_{\theta}(x) x^2 dx = E_q[x^2 \nabla_{\theta} \log q_{\theta}(x)]$$

For our example where $q_{\theta}(x) = N(\theta,1)$, this method gives $$\nabla_{\theta} E_q[x^2] = E_q[x^2 (x-\theta)]$$

Reparameterization trick is a way to rewrite the expectation so that the distribution with respect to which we take the expectation is independent of parameter $\theta$. To achieve this, we need to make the stochastic element in $q$ independent of $\theta$. Hence, we write $x$ as $$x = \theta + \epsilon, \quad \epsilon \sim N(0,1)$$

Then, we can write $$E_q[x^2] = E_p[(\theta+\epsilon)^2]$$ where $p$ is the distribution of $\epsilon$, i.e., $N(0,1)$. Now we can write the derivative of $E_q[x^2]$ as follows $$\nabla_{\theta} E_q[x^2] = \nabla_{\theta} E_p[(\theta+\epsilon)^2] = E_p[2(\theta+\epsilon)]$$