1
$\begingroup$

for example let us consider following table

enter image description here

know that entropy of variable is maximum when it is equally distributed,all of it's variable has equal probability,but what about joint entropy or conditional entropy?we know that channel capacity is equal

$I(X,Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)$

it is equal maximum when $H(X)$ is maximum and $H(X|Y)$ is minimum,but when it happens this?for first we can say that all marginal probability of $X$ should be equal to each other,but what about second?is it same also?clearly if $H(X|Y)$ is zero in addition of $H(X)$ be maximum,then we will have maximum mutual information,but when it happens?thanks in advance

$\endgroup$
1
$\begingroup$

$$H(X|Y)=\sum_{i,j}p(x,y)\log(p(x|y))$$

So to make this zero we want $p(x|y)=1$ whenever $p(x,y)\neq 0$. This means that $x$ is a function of $y$. So to maximise $I(X,Y)$, take both variables to be equal and with uniform marginals.

$\endgroup$
  • $\begingroup$ by uniform marginal you mean? $\endgroup$ – dato datuashvili Jun 27 '14 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.