So I guess some preface is that I'm working in machine learning, specifically variational autoencoders which are trying to approximate a distribution representing $P(x|z)$, which is the probability of x given latent variables z. So specifically in my case the probability that "Bwob" is mispelled given that we observe a B, w, o and b. So currently I have it so that at each index is a categorical distribution representing the probability it's some character at index i conditioned on other observed variables, but I'm wondering if I can use a gaussian distribution instead? (Reasons being it's much easier to make approximations to the parameters and by CLT if the data is i.i.d then the distribution will be approximately normal, also continuous RV are easier to integrate in probabilistic programming). So currently I have the distributions setup so I sample from a categorical, is there any mathematical grounds to having them be a gaussian, then rounding the sampled value to the nearest integer(Each integer representing the letter it could potentially be at index i, with A=0, B=1, etc)?
If I understand what you're doing correctly -- and mind you, I might not -- then you'd be imposing a bell curve of sorts to the alphabet. But this almost certainly isn't a reasonable model for the likelihood of various letters, which should look more like this (for example):
A good model of probabilities of letters should have spikes at common letters such as E, T, A, O, I, N, S, R and dips at infrequent letters like K, X, J, Q, Z. There is no reasonable way to impose a bell curve shape and accomplish all those things. The fundamental problem is that letters just aren't ordered in a way that has the likeliest ones in the middle of the alphabet and the least likely ones towards the ends.
That said, if you wanted to change your assignment of numbers to letters, you might be able to make it work. Assigning A to 0, B to 1, etc. suffers from the problems I outlined above -- but perhaps you could find your likeliest letters in your categorical distribution and assign then numbers in the middle of the 0-25 range. For example, you might choose 13 to be E, 12 to be T, 14 to be A, and so forth. Here, the ordering of the numbers is imposed entirely by which letters you want to be the likeliest ones, so the ordering of the alphabet itself is necessarily lost entirely. You might then be able to fit a Gaussian curve to this histogram; you'd really need to plot it, with the bell curve, to see if it does a good job modeling the distribution or not.
As a final note, I do strongly suspect that you're overstating the benefits of switching to a continuous distribution. It's not clear to me why parameter estimation would be easier, I have concerns about the CLT applying here at all, and discrete variables don't need to be integrated at all (just summed).
Please feel free to push back if I've misunderstood something and I'll edit (or delete) my response accordingly.