2
$\begingroup$

I have a random complex column vector $\mathbb{x}$ of length $L$ which has circularly symmetric complex gaussian probability density function with mean $0$ and covariance matrix $\sigma^2 \mathbb{I}$ where $\mathbb{I}$ is the identity matrix of size $L$. I have read that for such random vector the differential entropy is given as $$H(\mathbb{x})=\log_2 \det(\pi e\sigma^2\mathbb{I}).$$ I the formula for finding the entropy is as follows $$H(\mathbb{x})=-\int_{-\infty}^{\infty}p(\mathbb{x})\log_2\left(p(\mathbb{x})\right)d\mathbb{x}.~~~~~~\text{Eq. 1}$$ Further, I know that $$p(\mathbb{x})=\frac{1}{\pi^L \det(\sigma^2\mathbb{I})}\exp(-\frac{\mathbb{\|x\|^2}}{\sigma^2}).~~~~~\text{Eq. 2}$$ When I put Eq. 2 into Eq. 1 I get $$H(\mathbb{x})=\frac{1}{\sigma^2 \pi^L \ln(2)}\left[\int_{-\infty}^{\infty}\ln(\pi^L \sigma^2)\exp(-\frac{\|\mathbb{x}\|^2}{\sigma^2})d\mathbb{x}+\frac{1}{\sigma^2 }\int_{-\infty}^{\infty}\|\mathbb{x}\|^2\exp(-\frac{\|\mathbb{x}\|^2}{\sigma^2})d\mathbb{x}\right].$$ How to proceed further to achieve $H(\mathbb{x})=\log_2\det(\pi e \sigma^2)$. Any help in this regard will be much appreciated. Thanks in advance.

$\endgroup$
6
  • 1
    $\begingroup$ Convert the circularly symmetric integral over $L$ dimensions into a one-dimensional integral over a radius variable $r$, as shown here: youtube.com/watch?v=nqNzKeVCYBU. $\endgroup$ Commented Jun 15, 2018 at 0:35
  • $\begingroup$ @DavidG.Stork Thank you for your comment but that part I do not know how to do. Can you please add some steps as an answer? I will be grateful to you. $\endgroup$ Commented Jun 15, 2018 at 0:37
  • $\begingroup$ @DavidG.Stork do you mean that I should write $$\int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty}\exp(-\frac{|x_1|^2+\cdots |x_L|^2}{\sigma^2})d|x_1|\cdots d|x_L|=\int_{-\infty}^{\infty}\exp(-\frac{L|r|^2}{\sigma^2})d|r|$$? Is this what you meant? $\endgroup$ Commented Jun 15, 2018 at 0:47
  • $\begingroup$ Related post #1: math.stackexchange.com/questions/3914819/… $\endgroup$
    – develarist
    Commented Nov 24, 2020 at 5:17
  • $\begingroup$ Related post #2: math.stackexchange.com/questions/3906538/… $\endgroup$
    – develarist
    Commented Nov 24, 2020 at 5:17

2 Answers 2

2
$\begingroup$

Formally, the entropy $H(X)$ of a complex random variable $X$ is defined as the entropy $H(\Re(X),\Im(X))$ of the (vector) random variable $[\Re(X),\Im(X)]$, consisting of the real and imaginary components of $X$. (This is in accordance to how the pdf of a complex variable is defined.) Now, for the case of $X$ being circularly symmetric Gaussian of zero mean and covariance $\sigma^2 \mathbf{I}$, its real and imaginary components are i.i.d., Gaussian of zero mean and variance $(\sigma^2/2)\mathbf{I}$. Therefore,

$$ \begin{align} H(X) &= H(\Re(X),\Im(X)) \\ &= H(\Re(X)) + H(\Im(X))\\ &= \frac{1}{2} \log \det \left(2\pi e \frac{\sigma^2}{2} \mathbf{I} \right) + \frac{1}{2} \log \det \left(2\pi e \frac{\sigma^2}{2} \mathbf{I} \right)\\ &= \log \det \left(\pi e \sigma^2 \mathbf{I} \right) \end{align} $$

$\endgroup$
2
  • $\begingroup$ Thank you so much for your answer. But I do not even know how $H(\mathbb{R(X)})=\frac{1}{2}\log \det(2\pi e\frac{\sigma^2}{2}\mathbb{I})$. Can you please provide a reference where I can find a step by step proof of this. Actually I tried to search over internet and I only find the final result and not the step by step proof. Therefore, I posted this question. $\endgroup$ Commented Jun 15, 2018 at 8:17
  • $\begingroup$ @FrankMoses It follows by direct computation of the definition. You may treat it as an exercise or search for it, e.g., in online course notes, as it is a standard derivation $\endgroup$
    – Stelios
    Commented Jun 15, 2018 at 8:54
1
$\begingroup$

I am not sure if you have already gotten the answer. I am just posting my method. Hope it helps!

  • First let us derive the entropy for a real random vector $\mathbf{Z}\in \mathbb{R}^{L\times1}$ which has multivariate gaussian pdf of mean $0$ and a generic covariance matrix $\mathbf{\Sigma}$ :

$\rightarrow$ Note that the pdf in this case ($\because \mu=0$) is given by :

$$f_{\mathbf{Z}}\left(z_{1}, \ldots, z_{L}\right)=\frac{\exp \left(-\frac{1}{2}\mathbf{z}^T\boldsymbol{\Sigma}^{-1}\mathbf{z}\right)}{\sqrt{(2 \pi)^{L}det(\boldsymbol{\Sigma})}}$$

$\rightarrow$ The differential entropy is given by :

$ \begin{align} H(\mathbf{Z})&=-\int_{-\infty}^{\infty} f_\mathbf{Z}(\mathbf{z}) \log _{2}(f_\mathbf{Z}(\mathbf{z})) d \mathbf{z}\\ &= \frac{1}{2} \log _{2}\left((2 \pi)^{L} det(\boldsymbol{\Sigma})\right)\int_{-\infty}^{\infty} f_\mathbf{Z}(\mathbf{z}) d \mathbf{z}-\frac{1}{2} \log _{2}(e)\int_{-\infty}^{\infty} f_\mathbf{Z}(\mathbf{z})[\mathbf{z^T\Sigma^{-1} z}] d \mathbf{z}\\ &=\frac{1}{2} \log _{2}\left(det(\left(2 \pi \mathbf{\Sigma}\right)\right)+\frac{1}{2} \log _{2}(e) \mathbb{E}\left[\mathbf{z}^T \mathbf{\Sigma}^{-1} \mathbf{z}\right] \\ &=\frac{1}{2} \log _{2}\left(det(\left(2 \pi \mathbf{\Sigma}\right)\right)+\frac{1}{2} \log _{2}(e)\mathbb{E}\left[tr\left(\mathbf{z}^T \mathbf{\Sigma}^{-1} \mathbf{z}\right)\right]\\ \text{[Using Trace Trick]}\\ &=\frac{1}{2} \log _{2}\left(det(\left(2 \pi \mathbf{\Sigma}\right)\right)+\frac{1}{2} \log _{2}(e)\mathbb{E}\left[tr\left(\mathbf{\Sigma}^{-1}\mathbf{z}^T \mathbf{z}\right)\right]\\ &=\frac{1}{2} \log _{2}\left(det(\left(2 \pi \mathbf{\Sigma}\right)\right)+\frac{1}{2} \log _{2}(e)\cdot tr\left(\mathbf{\Sigma}^{-1}\mathbb{E}\left[\mathbf{z}^T \mathbf{z}\right]\right)\\ &=\frac{1}{2} \log _{2}\left(det(\left(2 \pi \mathbf{\Sigma}\right)\right)+\frac{1}{2} \log _{2}(e)\cdot tr\left(\mathbf{\Sigma}^{-1}\mathbf{\Sigma}\right)\\ &=\frac{1}{2} \log _{2}\left(det(\left(2 \pi \mathbf{\Sigma}\right)\right)+\frac{L}{2} \log _{2}(e)\\ &=\frac{1}{2} \log _{2}\left(det(\left(2 \pi e \mathbf{\Sigma}\right)\right)\\ \end{align} $

  • Now as @Stelios pointed out, we can write the following (where $\mathbf{X}$ is a complex random vector) :

\begin{aligned} H(\mathbf{X}) &=H(\Re(\mathbf{X}), \mathfrak{I}(\mathbf{X})) \\ &=H(\Re(\mathbf{X}))+H(\mathfrak{I}(X)) \\ &=\frac{1}{2} \log\left(det\left(2 \pi e \frac{\mathbf{\Sigma}}{2}\right)\right)+\frac{1}{2} \log\left(det\left(2 \pi e \frac{\mathbf{\Sigma}}{2}\right)\right) \\ &= \log\left(det\left(\pi e \mathbf{\Sigma}\right)\right) \end{aligned}

  • For your specific case, where covariance matrix is $\sigma^2\mathbf{I}$, we get the required entropy expression by substituting this :

$$H(\mathbf{X}) = \log\left(det\left(\pi e \sigma^2\mathbf{I}\right)\right)$$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .