# Finding the Information Gain in sets

I have a universe, $$U = \{a, b, c, d, e, f\}$$ and sets $$A = \{a, b, c\}$$ and $$B = \{a, d, e, f\}$$

If $$P(A) = P(X = x \in A)$$ and $$P(B) = P(X = x \in B)$$, where $$X$$ is a random variable defined by uniformly selecting elements of $$U$$.

I have the following probabilities based on the above.

$$p(A) = 1/2$$

$$p(A,B) = 1/6$$

$$p(A|B) = 1/4$$

I know that entropy $$H(X) = - \sum(P(x_i) \log P(x_i))$$ and information gain is $$IG(X|Y) = H(X) - H(X|Y)$$

Calculating entropy in A, I get: $$H(A) = - 3(1/6) \log (1/6) = 0.389$$

Now I am having a hard time to compute $$IG (A | B)$$ (which is defined as $$H(A) - H(A|B)$$) as I don't know how to compute the $$H(A|B)$$ here. Any clue?

• Your notation is confusing. Traditionally, when one writes $H(X|Y)$ both $X,Y$ are random variables. When you write $H(X|A)$ ... what is $A$ ? It's surely not a set (as was defined) then it shohuld be either an event ($X \in A$) or ar indicator random variable for that event. You should use another letter , and clarify its meaning. – leonbloy Feb 10 at 13:51
• Hi @leonbloy, A & B are both random variables. when I wrote H(X|A) was just a generic term, but what I am looking for is H(A|B) to be able to compute information gain IG(A|B). – user102859 Feb 10 at 14:07
• Please, reread my comment. You are denoting by $A$ a set, an event, and (it seems now) a random variable. That's a mess. And I'm not being pedantic. $H(X|A)$ has different meanings when $A$ is an event and when it's a random variable. – leonbloy Feb 10 at 16:57
• Thanks. Modified the post. – user102859 Feb 11 at 4:12