# Calculating the number of unordered unique combinations of x cards to make specific hands

I'm currently in the process of writing an application related to poker, and I've been struggling to determine the full formula for a piece of this. I need to determine the number of ways to combine x cards (ignoring suit) where the best 5 card hand made from that combination is not a straight or high card hand. First piece. f(x, y) is r-com with repetition of x possible values for y cards, g(x, y) is r-com without repetition, i is the number of cards in a hand. Here's where I'm at so far:

f(13, i) - g(13, i) - f(13, i-5)

So that's all combinations of cards where at least one card is repeated at least once, but no card is repeated at least more than 4 times. However, I forgot to account for the fact that at 6 cards or more, it's possible to have both a straight and a pair. So I need to subtract those. Here's what I've got so far for that:

9*(f(12, i-5) - g(7, i-5))
+
(f(13, i-5) - g(8, i-5))

However, this number also doesn't work for 8 or more cards, as the second part of the formula includes cases where you have both a straight and a full house, four of a kind, or an illegal hand (5 or more of a card). And that's where I'm stuck. Any help would be very much appreciated!

-
Are you looking for the number of hands that have at least one pair? You don't say whether you count hands with a flush (five of a suit). Alternately, if there is a straight, does there have to be a higher hand, such as a flush, full house, or four of a kind? – Ross Millikan May 8 '12 at 13:22
Flushes are irrelevant as I'm working in a way that ignores suit. And yes, that is what I'm trying to exclude from the second formula are hands where there is a straight and a higher hand (full house, four of a kind, and illegal hands which it also includes right now) – sgrif May 9 '12 at 13:42

It sounds like you want the number of hands that have no pairs as all full houses, four of a kinds, etc. have one. The number of $n$ card hands that have no pair is $52 \cdot 48 \cdot 44 \ldots (56-4n)$ as each card you draw prevents three more from being added.