1
$\begingroup$

Actually extremum, my question is whether my procedure is corrent or not.

According to wikipedia the maximum entropy distribution constrained by:

$$ \int_{-\infty}^{\infty}f_j(x)p(x)dx=a_j $$

has the form $$ \ln{p(x)} = -1 + \lambda_0 + \sum_{j=1}^{n}\lambda_j f_j(x) $$

This result relies on differential entropy, which has some undesirable properties. From wikipedia:

However, differential entropy does not have other desirable properties:

  1. It is not invariant under change of variables, and is therefore most useful with dimensionless variables.
  2. It can be negative.

A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).

So I set up to Finding $p(x,y)$, $M_x(x)$ and $M_y(y)$ that minimize $I(X;Y)$:

\begin{equation}\label{mi_continuous_mutual_information_definition} I(X;Y) = \int dx \int dy \, p(x,y) \log \frac{p(x,y)}{M_x(x)M_y(y)} \end{equation}

subjected to

\begin{eqnarray} \int f_j(x,y)p(x,y)dx dy=a_j \\ \int p(x,y)dx dy=1 \\ \int M_x(x)p(x,y)dx dy=a_x\\ \int M_y(y)p(x,y)dx dy=a_y \end{eqnarray}

With the last two constrains I intend that the scalar product of the marginal of $p(x,y)$ with $M_x(x)$ is an extremum. Following this pdf

Following the proof in wikipedia using Lagrange Multipliers, I consider the functional:

\begin{equation} \begin{split} J\left[p(x,y)\right] &= I(X;Y)\\ &- \int \lambda_0(x,y) (p(x,y) - 1) dx dy\\ &- \sum_j \int \lambda_j(x,y) (f_j(x,y) p(x,y) - a_j) dx dy \\ &- \int \lambda_x(x,y) (M_x(x) p(x,y) - a_x) dx dy \\ &- \int \lambda_y(x,y) (M_y(y) p(x,y) - a_y) dx dy \end{split} \end{equation}

Considering its derivatives w.r.t the marginals

\begin{eqnarray} \frac{\delta J\left[p(x,y)\right]}{\delta M_x(x)} &= 0 = \frac{p(x,y)}{M_x(x)} - \lambda_x(x,y) p(x,y)\implies \lambda_x(x,y) = \frac{1}{M_x(x)}\\ \frac{\delta J\left[p(x,y)\right]}{\delta M_y(y)} &= 0 = \frac{p(x,y)}{M_y(y)} - \lambda_y(x,y) p(x,y) \implies \lambda_y(x,y) = \frac{1}{M_y(y)} \end{eqnarray}

and, finally, w.r.t $p(x,y)$

\begin{equation} \begin{split} \frac{\delta J\left[p(x,y)\right]}{\delta p(x,y)} &= 0\\ &= \ln p(x,y) - \ln M_x(x) - \ln M_y(y) + 1 - \lambda_0(x,y)\\ &- \lambda_x(x,y) M_x(x) - \lambda_y(x,y) M_y(y)\\ &- \sum_j \lambda_j(x,y) f_j(x,y) \end{split} \end{equation}

solving for $\ln p(x,y)$ and replacing $\lambda_y(x,y)$ and $\lambda_x(x,y)$:

\begin{equation} \begin{split} \ln p(x,y) &= \ln M_x(x) + \ln M_y(y) + \lambda_0(x,y)\\ &+ \sum_j \lambda_j(x,y) f_j(x,y) + 1 \end{split} \end{equation}

This shows the extremum of mutual information.

Is this derivation correct?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.