i want to know when it is equal channel capacity or $I(X,Y)$ maximum or

enter image description here where

$I(X,Y)=H(X)-H(X\mid Y)=H(Y)-H(Y\mid X)$

now if we have two random variable with some specific distribution function,we can calculate it's mutual information easily right?but for getting maximum one,we may change their probability distributions,but of course question arises :how many times we should change it?there should be right some limit,for example we have following table

enter image description here

with some probability distribution functions,we can calculate for instance

$H(Y\mid X)$ ,actually it is calculated so we get $13/8$,so we have

$I(X,Y)=H(Y)-H(Y\mid X)=2-13/8=3/8$

but is it maximum?or how can i calculate maximum mutual information?should i assign different probabilities or?thanks in advance


The capacity of a channel depends, not on the input of the output but on the channel. In the standard probabilistic model, with memoryless property, the channel determines the conditional (transition) probabilities $P(Y|X)$ ; furthermore, if both the input $X$ and the output $Y$ have finite alphabets, the conditional probabilities can be represented as a matrix (transition matrix).

Hence, given (fixed) the transition probability matrix $P(Y|X)$, for each possible probability distribution of the input $P(X)$ you can obtain the joint $P(X,Y)$ as well as the output marginal $P(Y)$ - and from that, you can compute the mutual information $I(X,Y)$. The task is to find, for all possible $P(X)$, the one which maximizes $I(X,Y)$.

  • $\begingroup$ yes i know,for instance entropy of given variable is maximum when all it's variable has equal probbaility,what about conditional or joint entropy? $\endgroup$ – dato datuashvili Jun 26 '14 at 18:54
  • $\begingroup$ When you vary $P(X)$ you vary both $P(Y)$ and $P(Y|X)$, so the mutual information changes. You cannot say much more in general, the maximization is not trivial except in particular cases. I'm not sure which is your problem with this. $\endgroup$ – leonbloy Jun 26 '14 at 19:04
  • $\begingroup$ i see,ok thanks in advance,that means that channel capacity we can't calculate directly numerical like this yes?we should have some concrete model or system $\endgroup$ – dato datuashvili Jun 26 '14 at 19:05
  • $\begingroup$ Yes, you should try to calculate the capacity of some simple channels (the exercises of any textbook) to get the idea. Eg fi.muni.cz/~xbouda1/teaching/2010/IV111/lecture9.pdf For the general case (arbitrary transition matrix) it must be done numerically. $\endgroup$ – leonbloy Jun 26 '14 at 19:10
  • $\begingroup$ thanks in advance,now it is not topic for our class,it is package course so we can't study it in one week thanks for your efford $\endgroup$ – dato datuashvili Jun 26 '14 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.