Given two discrete probability distributions over $n$ events, with $p_i$ and $q_i$ denoting the probability that the ith event occurs respectively, I am looking for an upper bound of the following expression: $$ \left|\sum_{i=1}^n p_i (\log^2 p_i - \log^2 q_i) \right|. $$ This is very similar to the KL-divergence, with the exception that the logs are squared. This innocent change, however, seems to destroy the standard ways of proving the properties of the KL-divergence via Log Sum Inequality, Gibb's inequality, etc. and I haven't been able to figure out a clever workaround. Note that I'm only interested in bounding the magnitude of the expression but I'm not even sure it's always positive.

For my purposes, it would be good enough to have an upper bound like $\leq c(n) D(p|q)^2$, where $D(p|q)$ denotes the KL-divergence and $c(n)$ is a constant that may depend on $n$.

Finally, I should also note that I need to know the answer only for the special case in which $p$ is a joint distribution over two random variables and $q$ is the product of its marginals (i.e. the special case in which the KL-divergence gives the mutual information of $p$), maybe that special case is easier to handle.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.