# Conditional entropy of continuous and discrete random variable vectors

I have a continuous, multivariate normal distributed random variable vector $\mathbf{x} \sim \mathcal{N}(0, \mathbf{\Sigma} )$, $\mathbf{\Sigma} \in \mathbb{R}^{N \times N}$, and a discrete random variable vector $\mathbf{r} = [r_1, \ldots,r_N]^T$. Furthermore, $r_i$ is defined as $$r_i = \begin{cases} -l &,\text{for } x_i \in (-\infty,0)\\ \hphantom{-}l &, \text{otherwise} \end{cases} \quad, i \in \{1,\ldots,N\}.$$

My goal is to calculate the differential entropy $h(\mathbf{x}\vert\mathbf{r})$, where I'm actually not sure if it is still a differential entropy because $\mathbf{x}$ is conditioned on the discrete $\mathbf{r}$.

In (1) the conditional differential entropy is defined as $$h(X \vert Y ) = - \int f(x,y) \, \log ( f(x,y)) \, dx \,dy.$$

Reference (2) discusses similar problems. As a result, I defined my conditional differential entropy as

$$h(\mathbf{x}\vert\mathbf{r}) = - \sum_{i \in \mathcal{R}} \int_{\mathbb{R}^N} p( \mathbf{r}_i \vert \mathbf{x}) \, f(\mathbf{x}) \, \log \left( \frac{ p( \mathbf{r}_i \vert \mathbf{x}) \, f(\mathbf{x})}{ p(\mathbf{r}_i) } \right) \, d\mathbf{x}$$

where $\mathcal{R}$ denotes the set of all possible realizations of $\mathbf{r}$, $p(\mathbf{r}_i)$ denotes their probability, $p( \mathbf{r}_i \vert \mathbf{x})$ denotes the conditional probability of $\mathbf{r}_i$ given $\mathbf{x}$ and $f( \mathbf{x} )$ denotes the PDF of $\mathbf{x}$. Furthermore, I define $p( \mathbf{r}_i)$ as

$$p( \mathbf{r}_i \vert \mathbf{x}) = \begin{cases} 1 &, \mathbf{x} \in I_i\\ 0 & \text{otherwise} \end{cases},$$

where $I_i$ denotes some appropriately defined $N$ dimensional interval. Using this it follows

$$h(\mathbf{x}\vert\mathbf{r}) = - \sum_{i \in \mathcal{R}} \int_{I_i} f(\mathbf{x}) \, \log \left( \frac{ f(\mathbf{x})}{ p(\mathbf{r}_i) } \right) \, d\mathbf{x}.$$

Questions

1. Are my definition of $h(\mathbf{x}\vert\mathbf{r})$ and $p( \mathbf{r}_i \vert \mathbf{x})$ correct?
2. How can I solve this integral? It seems to be related to the entropy of the multivariate normal distribution. Can I somehow break it down into the $x_i$s? A numerical solution would also be fine for me. However, many-dimensional numerical integration seems to be challenging, too.

References

(1) Cover, Thomas M., and Joy A. Thomas. Elements of information theory. John Wiley & Sons, 2012.

(2) Nair, Chandra, Balaji Prabhakar, and Devavrat Shah. "On entropy for mixtures of discrete and continuous variables." arXiv preprint cs/0607075 (2006). Available: http://chandra.ie.cuhk.edu.hk/pub/papers/manuscripts/ENT-arx07.pdf

By the chain rule of entropy, it holds

$$\tag{1} h(\mathbf{x},\mathbf{r})=h(\mathbf{r})+h(\mathbf{x}\mid \mathbf{r})$$

as well as

$$\tag{2} h(\mathbf{x},\mathbf{r})=h(\mathbf{x})+h(\mathbf{r}\mid \mathbf{x}).$$

Now, since $\mathbf{r}$ is a deterministic function of $\mathbf{x}$, it holds

$$h(\mathbf{r}\mid \mathbf{x})=0,$$

which, after substituting in (2) gives

$$h(\mathbf{x},\mathbf{r}) = h(\mathbf{x}).$$

Substituting the last result in (1) leads to

\begin{align} h(\mathbf{x}\mid \mathbf{r}) &= h(\mathbf{x})-h(\mathbf{r})\\ &=\frac{1}{2}\log\left((2\pi e)^N \det(\mathbf{\Sigma})\right)-\left(-\sum_\mathbf{r}p(\mathbf{r})\log(p(\mathbf{r}))\right). \end{align}

I am not aware of any closed-form formula for $p(\mathbf{r})$. According to this answer, it does not have one. Therefore, you would have in principle to enumerate all the possible ($2^N$) vectors $\mathbf{r}$ and numerically compute the pmf $p(\mathbf{r})$. My guess is that this procedure will become impractical for moderate $N$.

Remark: The above formula is the same as yours. Indeed,

\begin{align} -\sum_{i\in \mathcal{R}} \int_{I_i} f(\mathbf{x})\log\frac{f(\mathbf{x})}{p(\mathbf{r}_i)}d\mathbf{x}&=-\sum_{i\in \mathcal{R}} \int_{I_i} f(\mathbf{x})\log f(\mathbf{x})d\mathbf{x}- \left(-\sum_{i\in \mathcal{R}} \log p(\mathbf{r}_i)\int_{I_i} f(\mathbf{x}) d\mathbf{x}\right)\\ &=-\int_{\mathbb{R}^N} f(\mathbf{x})\log f(\mathbf{x})d\mathbf{x}- \left(-\sum_{i\in \mathcal{R}} p(\mathbf{r}_i) \log p(\mathbf{r}_i)\right), \end{align}

since $\cup_{i\in\mathcal{R}}I_i=\mathbb{R}^N$, $I_i \cap I_j = \emptyset, \forall i\neq j$, and $\int_{I_i} f(\mathbf{x}) d\mathbf{x} = p(\mathbf{r}_i)$.

• Thank you very much, that was really helpful. I agree, computing $p(\mathbf{r})$ is still an issue. I'll figure out what to do about it. – Enzo Jul 18 '17 at 7:58

How do you justify $$h(x|r) = h(r) + h(r|x)$$? Is $$h(r)$$ the differential entropy of $$r$$? Isn't $$r$$ a discrete random variable?