0
$\begingroup$

Suppose I have a random vector $\bar{X}=[X_{1},X_{2}, X_{3}]$. X1,X2 and X3 can take values from the alphabet {0,1,2,3} . (This can be even generalized to a finite set of cardinality N). I don't have any information on the probability distribution of these random variables.

If X is random variable then its entropy is:

$H(X) = -\displaystyle\sum_{x} p(x)\log p(x).$

Similarly, I understand that we have definitions for joint entropy for the random vector $\bar{X}=[X_{1},X_{2}, X_{3}]$.

I understand that If no constraints exist the entropy maximizing distribution is uniform distribution. So here the joint pdf will be uniform pdf when no constraints exist.

My questions:

  1. What if I have a constraint on $X_{3}$ such that $E[{X_{3}}^{2}] <\alpha$
  2. What if I have a constraint on $X_{2}$ and $X_{3}$ such that $E[{X_{2}}^{2}] +E[{X_{3}}^{2}]<\alpha$

How will we go about finding the entropy maximising distributions in this case? Can anyone give some details.

Thanks.

$\endgroup$
  • $\begingroup$ read into convex optimization $\endgroup$ – LinAlg Oct 10 '18 at 14:41
  • $\begingroup$ Related (similar) question math.stackexchange.com/questions/2949210/… $\endgroup$ – leonbloy Oct 10 '18 at 20:13
  • $\begingroup$ I think you need to say something about de domain of your variables. Otherwise, your question seems pointless. Entropy is not really related to the values of your variable but to the incertainty of this value. Furthermore, if you give no constraint on your distributions, then entropy has no limit (entropy of a uniform distribution on a segment of length N is log(N)) $\endgroup$ – Arnaud Mégret Oct 26 '18 at 14:13
  • $\begingroup$ @ArnaudMégret X1,X2 and X3 can take values from the alphabet {0,1,2,3} . This can be even generatilized to finite set of cardinnality N. I don't have any information on the probability distribution of these random variables. $\endgroup$ – Jyotish Robin Oct 26 '18 at 14:17
  • $\begingroup$ you should edit your question to add this information. $\endgroup$ – Arnaud Mégret Oct 26 '18 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.