Statistic to compare multiple discrete empirical distributions; no sample counts Can you suggest an appropriate analysis method?
Data I’m trying to analyze:

*

*A bunch of related discrete probability distributions $P_i(x)$. These distributions are empirical but they are not generated from samples. (Instead, they are generated in one shot with a deep neural network.)

*Each distribution $P_i(x)$ is across $N_i$ classes, where in general $N_i \neq N_{i+1}$; the "null hypothesis" that each distribution is the discrete uniform distribution, i.e. $P_i(x_j) = 1/N_i$. (Putting "null hypothesis" in quotes here since it may be an abuse of terminology.)

*I want to compute a statistic independent of the number of classes in each distribution that records how “surprising” a given probability $P_i(x_j)$ is given the null hypothesis. I’d like this statistic to be comparable across distributions $P_i$.

For example, suppose I have two empirical distributions $P_1$ and $P_2$:
P_1: {class A: 0.33, class B: 0.67}
P_2: {class C: 0.33, class D: 0.33, class E: 0.34}

I don't know anything about these distributions except that I originally expected them to be uniform. Clearly, $P_1$ is less uniform than $P_2$, but how do I quantify that independent of the number of classes in each distribution? The statistics for $P_1$ should be further from their null-case value than for $P_2$.
I don't have a quantitative way of describing how much deviation a 2-class distribution requires to be equivalent to the deviation in a 10-class distribution, and am open to suggestion / best practice.
My objective is not to perform a hypothesis test per se but instead to efficiently find extreme $P_i(x_j)$.
It doesn't seem like the Chi-squared statistic is applicable out of the box since I don't have sample counts, but, again, open to suggestion.
The Kolmogorov-Smirnov test seemed to generic and doesn't look like it yields per-$x_j$ statistics.
EDIT
Additional clarification: My question is akin to someone supplying a bunch of discrete distributions without any additional context and requesting, "please rank these distributions in order of how non-uniform they are, and tell me which elements of each led you to that conclusion."
 A: Tenative "answer" addressing notational/terminology issues relating to question
If anyone has any insights into a "correct" method for comparing "uniformness" that would be welcome, but the idea of distances between functions as proposed by the OP seems sensible.
Let $F_{U}$ be the CDF of a random variable $U$ with discrete uniform distribution with support set $\{a_{1},...,a_{m}\}$, i.e. $m$ support points with $Pr[U=a_{j}]=m^{-1}$ for all $j=1,...,m$, where $Pr[U\leq u]=F_{U}(u)=(\lfloor u\rfloor-a_{1}+1)/(a_{m}-a_{1}+1)$ for $u\in[a_{1},a_{m}]$.
$F_{n}(x)$, the discrete uniform measure, is defined as $F_{n}(x):=n^{-1}\sum_{i=1}^{n}1\{x_{i}\leq x\}$ for some sample $X_{1}=x_{1}$,...$X_{n}=x_{n}$ (regardless of which random variable $X$ say, the sample is a sample of).
Then $F_{n}\equiv F_{U}$ when $n=m$ and when the sample contains $n$ unique values since when $\sum_{i}^{n}1\{x_{i}\leq x\}$ increases it will jump by precisely one as $x$ runs through $\mathbb{R}$. Conversely even if $n=m$, if the sample contain just $2$ unique values say, then $F_{n}$ will have just two jumps (at $b_{1}$ and $b_{2}$ say) and so $F_{n}\equiv F_{U}$ cannot occur if $m>2$ since $F_{n}(b_{1})=0.5$ and $F_{n}(b_{2})=1$ but $F_{U}(a_{1})=m^{-1}<0.5$. Thus although the sample could be a sample from a continuous random variable, $F_{n}$ will have discrete jumps whose number will not exceed $n$ (and will be equal to $n$ if the sample contains no duplicates).
When you speak of "classes" I think you mean a random variable $X$ having discrete law $\mu_{X}$ with support set $S_{X}=\{a_{1},a_{2},a_{3}\}$ say. Then $Pr[X=a_{i}]=p_{i}=\mu_{X}(\{a_{i}\})$ for $i=1,2,3$. But the empirical distribution $F_{n}$ of a sample $X_{j}=x_{j}$ for $j=1,2,...,n$ is not the same thing as the distribution function for $X$ unless $x_{j}\in S_{X}$ for all $j$ and $\sum_{j=1}^{n}1\{x_{j}\leq a_{i}\}=\sum_{a\in S_{X}}1\{a\leq a_{i}\}\mu_{X}(\{a\})$ for all $i=1,2,3$. We can use the notation $F_{X}(a_{i})=\sum_{a\in S_{X}}1\{a\leq a_{i}\}\mu_{X}(\{a\})$ to be notationally similar to the continuous random variable case where the distribution function $F$ is different from the law $\mu$ (i.e. the distribtuion), with the understanding $\mu_{X}(\{x\})=0$ for any $x\not\in S_{X}$.
The terminology is only a distraction here since it seems you know what you want to do - it seems the KL-divergence or something like this is along the right lines. For instance
$$||F_{n}-F_{U}||=\underset{x\in\mathbb{R}}{sup}|F_{n}(x)-F_{U}(x)|$$
is the uniform distance between $F_{n}$ and $F_{U}$ and
$$||F_{X}-F_{U}||=\underset{x\in\mathbb{R}}{sup}\left|F_{X}(\{x\})-F_{U}(x)\right|$$
is the uniform distance between $F_{X}$ and $F_{U}$. But remeber $F_{n}$ could have hundreds of jumps and $F_{X}$ only 3 for example - i.e. $F_{n}$ can be very "jumpy". However the concept of distances between functions is I is good thinking.
