I have been reading a lot about var. inference, yet every case ends with terms and derivations, what I miss is a broad picture of a problem. Please confirm if I understand correctly:

Let's assume I live in a house with a dog. My dog barks, sometimes for no reason, but mostly if someone is in front of the house. As I don't know if someone is in front of the house, I use data from my neighbour who calls me once per day and and tells me if she saw any people near to my house.

And this is where inference comes to the game.

My model is:


P(X) = someone in front of the house

P(Z) = Dog is barking

Q(Z|X) = Neighbour's data about people in front of my house when dog was barking

I would like to know:


So here comes the KL divergence:

Which basically mean, do relative distance of neighbour's data when she saw a person, given dog barking vs. my observation of dog barking and people being in front of the house. Is this it, is my explanation correct?


The point of variational inference is a technique for making Bayesian computation more efficient, but the logic spoke of in the OP is just general Bayesian principles of calculating the posterior, which is not the key insight or goal of variational inference.

The goal is largely one of using optimization to approximate hard-to-estimate distributions. Namely, let us choose some family of approximate densities $L$, which can be Gaussians or some other distribution. Then we try to find a member of the family that minimizes the KL-divergence to a posterior. Namely, we want $$q^*(z) = \arg \min_{q(z) \in L} \text{KL} (q(z) || p(z|x))$$

We then use this member $q^*(z)$ to calculate the interested posterior information. So what we are basically doing is trying to find a member of our family, like a Gaussian, that comes close to the distribution of interest.

For more information, read the review by Blei, Kucukelbir and McAuliffe.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.