# When does entropy bound cross-entropy?

Suppose $p$ and $q$ are discrete distributions over $k$ outcomes, $q$ is given, for which $p$ does the following hold?

\begin{equation*} p_1 \log q_1 + \ldots + p_k \log q_k \ge q_1 \log q_1 + \ldots + q_k \log q_k\end{equation*}

-

p such that:

$$\prod_i q_i^{p_i-q_i} \ge 1$$

I arrive at this point with the following calculations:

First of all I rewrite the equation:

$$\sum_i p_i \log q_i \ge \sum_i q_i \log q_i$$

And

$$\sum_i (p_i \log q_i - q_i \log q_i) = \sum_i (\log q_i^{p_i-q_i}) = \log \prod_i q_i^{p_i-q_i} \ge 0$$

The most simple deduction about p is that

$$p_i \ge q_i , \forall i$$

with

$$q_i \ge 1$$

-
Welcome to math.SE! We support TeX equations here, but they have to be surrounded by dollar signs. See meta.math.stackexchange.com/questions/107/… for detail. – kennytm Aug 20 '10 at 12:34