# Normalizing a conditional probability to within range of a Sigmoid function

Given the following scenario from another post of mine where we are building a matrix that expresses the probability of first order transitions from one character to another in an english text.

We take a book, and count the number of times the letter 'e' occurs in that book -- say 15,000. Then we count the number of times the next letter is 'f' -- say, 200. With this in hand, we put

$M(\text{'e'}, \text{'f'}) = 200/15000 = 1.33\%$.

Say instead we want to normalize this conditional probability to a range between 0 - 1, but discluding the absolute values 0 or 1 (only getting infinitesimely close to each extreme). Is there an accepted way to use a sigmoid function for this sort of normalization of a probability?

I don't know if this is a common practice, however, I think it would be useful in an AI application I am working on.

-
@Dan: The percent sign had gone missing because it wasn't escaped. – joriki Sep 15 '11 at 7:39
joriki, I see. I still don't understand the question because 200/15000 = 1.33% is already between 0 and 1. Maybe it is about using the sigmoid to normalize a likelihood ratio? – Dan Brumleve Sep 15 '11 at 7:42
@Dan: See my answer for how I understood the question to be intended. By the way, I don't get notified unless you put the '@' in front of the username. – joriki Sep 15 '11 at 20:10

If I understand correctly what you're trying to do, you're using the term "normalize" in a non-standard way. Conditional probabilities are already normalized in the sense that they add up to $1$.

I take it you want to map the relative frequencies observed in the text to conditional probabilities such that the relative frequencies in the closed interval $[0,1]$ are mapped to conditional probabilities in the open interval $(0,1)$. Note the plurals; you cannot just map one at a time as you seem to imply, since the conditional probabilities will not be conditional probabilities unless they add up to $1$, so their values have to be related.

Presumably the mapping should be monotonic in the sense that decreasing one relative frequency and increasing another should decrease and increase the respective conditional probabilities. In case by "infinitesimally close" you mean "arbitrarily close", the mapping cannot be done such that the resulting values get arbitrarily close to $0$ and $1$. This is precluded by monotonicity: The case where all relative frequencies are $0$ except one is $1$ has to be mapped to some conditional probabilities with a conditional probability less than $1$ corresponding to the relative frequency $1$, and by monotonicity no other set of relative frequencies can be mapped to a higher conditional probability than that.

However, if you merely wanted to allow, rather than require the conditional probabilities to get arbitrarily close to $0$ and $1$, there are many way to perform such a mapping. The most straightforward one would be to choose some monotonic function $f:[0,1]\to(0,1)$ and to define the conditional probabilities $p_i$ corresponding to the relative frequencies $\nu_i$ by

$$p_i=\frac{f(\nu_i)}{\sum_jf(\nu_j)}\;.$$

You could certainly use a sigmoid for $f$; I don't know of any "accepted" manner of choosing $f$.

-