So I guess some preface is that I'm working in machine learning, specifically variational autoencoders which are trying to approximate a distribution representing $P(x|z)$, which is the probability of x given latent variables z. So specifically in my case the probability that "Bwob" is mispelled given that we observe a B, w, o and b. So currently I have it so that at each index is a categorical distribution representing the probability it's some character at index i conditioned on other observed variables, but I'm wondering if I can use a gaussian distribution instead? (Reasons being it's much easier to make approximations to the parameters and by CLT if the data is i.i.d then the distribution will be approximately normal, also continuous RV are easier to integrate in probabilistic programming). So currently I have the distributions setup so I sample from a categorical, is there any mathematical grounds to having them be a gaussian, then rounding the sampled value to the nearest integer(Each integer representing the letter it could potentially be at index i, with A=0, B=1, etc)?

  • $\begingroup$ "by CLT if the data is i.i.d then the distribution will be approximately normal" -- what distribution are you talking about? The CLT applies to sample averages; if you're averaging something, then this seems fine, but it's not obvious to me how you're averaging anything here (and if it's not, the CLT simply does not apply). $\endgroup$ – Aaron Montgomery Dec 31 '19 at 15:13
  • $\begingroup$ averaging across data of names, sorry that was explicit. I'll have to average across names to get an approximate distribution yes? $\endgroup$ – user8714896 Dec 31 '19 at 16:16
  • $\begingroup$ I mean a good, old-fashioned, mean average. There should be some collection of $n$ things which you add and then divide by $n$. Is there such a collection? $\endgroup$ – Aaron Montgomery Dec 31 '19 at 16:19
  • 1
    $\begingroup$ with datasets definitely yea, but I think you cleared up a misconception I had about CLT $\endgroup$ – user8714896 Dec 31 '19 at 16:22

Probably not.

If I understand what you're doing correctly -- and mind you, I might not -- then you'd be imposing a bell curve of sorts to the alphabet. But this almost certainly isn't a reasonable model for the likelihood of various letters, which should look more like this (for example):

histogram of English letters

A good model of probabilities of letters should have spikes at common letters such as E, T, A, O, I, N, S, R and dips at infrequent letters like K, X, J, Q, Z. There is no reasonable way to impose a bell curve shape and accomplish all those things. The fundamental problem is that letters just aren't ordered in a way that has the likeliest ones in the middle of the alphabet and the least likely ones towards the ends.

That said, if you wanted to change your assignment of numbers to letters, you might be able to make it work. Assigning A to 0, B to 1, etc. suffers from the problems I outlined above -- but perhaps you could find your likeliest letters in your categorical distribution and assign then numbers in the middle of the 0-25 range. For example, you might choose 13 to be E, 12 to be T, 14 to be A, and so forth. Here, the ordering of the numbers is imposed entirely by which letters you want to be the likeliest ones, so the ordering of the alphabet itself is necessarily lost entirely. You might then be able to fit a Gaussian curve to this histogram; you'd really need to plot it, with the bell curve, to see if it does a good job modeling the distribution or not.

As a final note, I do strongly suspect that you're overstating the benefits of switching to a continuous distribution. It's not clear to me why parameter estimation would be easier, I have concerns about the CLT applying here at all, and discrete variables don't need to be integrated at all (just summed).

Please feel free to push back if I've misunderstood something and I'll edit (or delete) my response accordingly.

  • $\begingroup$ No your response is great. The reason it's easier is because we're using a neural net to learn the approximate distribution, but when the neural net with SVI does back propogation it's easier to differentiate and adjust the parameters of a continuous variable rather than a discrete one when minimizing the KL divergence. $\endgroup$ – user8714896 Dec 31 '19 at 16:19
  • $\begingroup$ Fair enough -- I don't know anything about neural networks, so I'll leave that evaluation to you. $\endgroup$ – Aaron Montgomery Dec 31 '19 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.