I have a universe, $ U = \{a, b, c, d, e, f\}$ and sets $A = \{a, b, c\}$ and $B = \{a, d, e, f\}$

If $P(A) = P(X = x \in A)$ and $P(B) = P(X = x \in B)$, where $X$ is a random variable defined by uniformly selecting elements of $U$.

I have the following probabilities based on the above.

$ p(A) = 1/2$

$p(A,B) = 1/6$

$p(A|B) = 1/4$

I know that entropy $H(X) = - \sum(P(x_i) \log P(x_i))$ and information gain is $IG(X|Y) = H(X) - H(X|Y)$

Calculating entropy in A, I get: $H(A) = - 3(1/6) \log (1/6) = 0.389$

Now I am having a hard time to compute $IG (A | B)$ (which is defined as $H(A) - H(A|B)$) as I don't know how to compute the $H(A|B)$ here. Any clue?

  • $\begingroup$ Your notation is confusing. Traditionally, when one writes $H(X|Y)$ both $X,Y$ are random variables. When you write $H(X|A)$ ... what is $A$ ? It's surely not a set (as was defined) then it shohuld be either an event ($ X \in A$) or ar indicator random variable for that event. You should use another letter , and clarify its meaning. $\endgroup$ – leonbloy Feb 10 at 13:51
  • $\begingroup$ Hi @leonbloy, A & B are both random variables. when I wrote H(X|A) was just a generic term, but what I am looking for is H(A|B) to be able to compute information gain IG(A|B). $\endgroup$ – user102859 Feb 10 at 14:07
  • $\begingroup$ Please, reread my comment. You are denoting by $A$ a set, an event, and (it seems now) a random variable. That's a mess. And I'm not being pedantic. $H(X|A)$ has different meanings when $A$ is an event and when it's a random variable. $\endgroup$ – leonbloy Feb 10 at 16:57
  • $\begingroup$ Thanks. Modified the post. $\endgroup$ – user102859 Feb 11 at 4:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.