1
$\begingroup$

As an example, assume that I have data $\mathbf{X}$, unobserved variables $\mathbf{Z}$ and model parameters $\pmb{\alpha}$, $\pmb{\beta}$. I am omitting the mentioning of any variational parameters. The variational distribution $q$ is over $\mathbf{Z}$, $\pmb{\alpha}$, and $\pmb{\beta}$. Also, let's say that the distribution over the latent variables contains an intractable normalization constant $\Omega(\pmb{\beta})$: \begin{align} p(\mathbf{Z} | \pmb{\beta}) = \frac{1}{\Omega(\pmb{\beta})}f(\mathbf{Z})\hat{p}(\mathbf{Z}|\pmb{\beta}) \end{align}

I now want to compute the ELBO: \begin{align} \text{ELBO}(q) &= \text{E}_q[\log p(\mathbf{X}, \mathbf{Z}, \pmb{\alpha}, \pmb{\beta})] - \text{E}_q[\log q(\mathbf{Z}, \pmb{\alpha}, \pmb{\beta})] \\ &= \text{E}_q[\log p(\mathbf{X}| \mathbf{Z}, \pmb{\alpha})] + \color{red}{\text{E}_q[\log p(\mathbf{Z}| \pmb{\beta})]} + \text{E}_q[\log p(\pmb{\alpha})] + \text{E}_q[\log p(\pmb{\beta})] - \text{E}_q[\log q(\mathbf{Z}, \pmb{\alpha}, \pmb{\beta})] \end{align}

My problem is with the term in red. I can try to decompose it as: \begin{align} \color{red}{\text{E}_q[\log p(\mathbf{Z}| \pmb{\beta})]} &= \text{E}_q[\log f(\mathbf{Z})] + \text{E}_q[\log \hat{p}(\mathbf{Z}| \pmb{\beta})] - \text{E}_q[\log \Omega(\pmb{\beta})] \end{align}

But how can I deal with $\text{E}_q[\log \Omega(\pmb{\beta})]$? I am using the ELBO as a termination criterion for variational Bayes. Since the ELBO is a lower bound on the evidence and I can assume $\Omega(\pmb{\beta}) > 0 \text{, }\forall \pmb{\beta}$, since it's a normalization constant, I am thinking of simply dropping this term. Any thoughts?

$\endgroup$
1
  • $\begingroup$ You're evaluating $\log\Omega(\beta)$, which may sometimes be negative. Do you have a specific (unnormalized) distribution $P(Z|\beta)$ that you're working with? I'm not sure there is a general solution to this type of problem, but there likely exist techniques for particular instances of it. $\endgroup$
    – Thoth
    Sep 10, 2020 at 1:13

1 Answer 1

0
$\begingroup$

So I think you may need to clarify some of your formulae.

If $\pmb\alpha$ and $\pmb\beta$ are parameters, then they shouldn't have probability distributions associated with them.

If $\pmb\beta$ is in fact a parameter, then you simply have,

$$\text{E}_q[\log\Omega(\pmb\beta)]=\log\Omega(\pmb\beta).$$

$\endgroup$
1
  • $\begingroup$ Thanks for the suggestion. I consequently updated the question description to emphasize that the variational distribution q is over $\mathbf{Z}$, $\pmb{\alpha}$, and $\pmb{\beta}$. $\endgroup$
    – JKB
    Sep 9, 2020 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.