I'm looking to create a variant of holdem allowing 6 and 7 card hands in special situations. Namely, 3 pair, 2 set, and quads over set (a full mansion).
My probability skills are rusty, but I'm still able to break down the problems, I think!
I've started trying to calculate the odds of 2-set so I can find where to rank the hand. I'm using the probability tree method. The branch at the top is clear, pocket pair vs mixed hand.
Pocket pair (0.059):
With a pocket pair, you need to hit a set with the card in your hand, and the board needs to hit a set. There are 12 other sets the board could hit besides the cards in your hand.
Odds of a permutation of such a board:
(2/50) * (4/49) * (3/48) * (2/47) * (44/46) = 8.3067e-06
The last value of 44/46 is because you want to preclude quads from the board. Using permutation with repetition, you get (5!/3!) or 20 for the permutations possible.
8.3067e-06 * 12 * 20 = 0.001993608 for the odds of 2 set when holding a pocket pair.
mixed hand (0.941)
When you don't have a pocket pair, the board needs to 2 pair both of your cards.
(3/50) * (2/49) * (3/48) * (2/47) * (44/46) = 6.2301e-06
Using permutation with repetition, you find there are (5! / (2! * 2!)) or 30 permutations of such a board.
6.2301e-06 * 30 = 0.000186903 of 2 set when you have a mixed hand
Combining both sides of the tree, you get
pocket pair: 0.059 * 0.001993608 = 0.0001176229
mixed hand: 0.941 * 0.000186903 = 0.0001758757
with a combined odds of 0.0002934986 or 0.0293%, ranking just below a straight flush (0.0279%).