# How to compute evidence lower bound (ELBO) when the complete log-likelihood is intractable?

As an example, assume that I have data $$\mathbf{X}$$, unobserved variables $$\mathbf{Z}$$ and model parameters $$\pmb{\alpha}$$, $$\pmb{\beta}$$. I am omitting the mentioning of any variational parameters. The variational distribution $$q$$ is over $$\mathbf{Z}$$, $$\pmb{\alpha}$$, and $$\pmb{\beta}$$. Also, let's say that the distribution over the latent variables contains an intractable normalization constant $$\Omega(\pmb{\beta})$$: \begin{align} p(\mathbf{Z} | \pmb{\beta}) = \frac{1}{\Omega(\pmb{\beta})}f(\mathbf{Z})\hat{p}(\mathbf{Z}|\pmb{\beta}) \end{align}

I now want to compute the ELBO: \begin{align} \text{ELBO}(q) &= \text{E}_q[\log p(\mathbf{X}, \mathbf{Z}, \pmb{\alpha}, \pmb{\beta})] - \text{E}_q[\log q(\mathbf{Z}, \pmb{\alpha}, \pmb{\beta})] \\ &= \text{E}_q[\log p(\mathbf{X}| \mathbf{Z}, \pmb{\alpha})] + \color{red}{\text{E}_q[\log p(\mathbf{Z}| \pmb{\beta})]} + \text{E}_q[\log p(\pmb{\alpha})] + \text{E}_q[\log p(\pmb{\beta})] - \text{E}_q[\log q(\mathbf{Z}, \pmb{\alpha}, \pmb{\beta})] \end{align}

My problem is with the term in red. I can try to decompose it as: \begin{align} \color{red}{\text{E}_q[\log p(\mathbf{Z}| \pmb{\beta})]} &= \text{E}_q[\log f(\mathbf{Z})] + \text{E}_q[\log \hat{p}(\mathbf{Z}| \pmb{\beta})] - \text{E}_q[\log \Omega(\pmb{\beta})] \end{align}

But how can I deal with $$\text{E}_q[\log \Omega(\pmb{\beta})]$$? I am using the ELBO as a termination criterion for variational Bayes. Since the ELBO is a lower bound on the evidence and I can assume $$\Omega(\pmb{\beta}) > 0 \text{, }\forall \pmb{\beta}$$, since it's a normalization constant, I am thinking of simply dropping this term. Any thoughts?

• You're evaluating $\log\Omega(\beta)$, which may sometimes be negative. Do you have a specific (unnormalized) distribution $P(Z|\beta)$ that you're working with? I'm not sure there is a general solution to this type of problem, but there likely exist techniques for particular instances of it. Sep 10, 2020 at 1:13

If $$\pmb\alpha$$ and $$\pmb\beta$$ are parameters, then they shouldn't have probability distributions associated with them.
If $$\pmb\beta$$ is in fact a parameter, then you simply have,
$$\text{E}_q[\log\Omega(\pmb\beta)]=\log\Omega(\pmb\beta).$$
• Thanks for the suggestion. I consequently updated the question description to emphasize that the variational distribution q is over $\mathbf{Z}$, $\pmb{\alpha}$, and $\pmb{\beta}$.