I'm doing a training exercise looking at 'test' policy claims database data where some of the policy data contains dummy dates (like a date that says it was taken out on 31 Dec 9999)

I'm planing on running a program over the data to find the most common dates used, and I'm expecting that it will bring back the dummy data, but I'd like to know in the case that the dummy data isn't something obviously wrong (say it was 1 April 1970) what the probability of the date and year being the same on $n$ records from a set of $m$ records, over a fixed range of years (say 100 years).

I'd try and look into this myself but I have no clue about probability. I couldn't find it on the wiki page nor in a check of the various birthday problem questions on this stack exchange.

  • $\begingroup$ In the usual birthday problem, we can assume by inspection of birth records that births are reasonably equidistributed over the year (this is not entirely true, but it is certainly not the case the half the people are born in May). But in your question, we cannot expect equidistribution at all because in a database of living people birthdays from the 1920s will be much rarer than birthdays from the 1970s. In a database of employees, birthdays in the 1990s will be much rarer than birth days in the 1980s and so on. $\endgroup$ – Phira Nov 27 '12 at 12:52
  • $\begingroup$ So, you either need to get some rough estimate of the probability distribution or you are happy with a very rough estimate of the probability in one direction that will be far from the true value. (Note that you might well have equidistribution if you study records of people who, in a particularly quiet part of the middle ages, died in childbirth in a particular country or something like that). $\endgroup$ – Phira Nov 27 '12 at 12:55
  • $\begingroup$ As this is test data I'm happy to make wide sweeping assumptions about the distributions of policy/birthdates. $\endgroup$ – Pureferret Nov 27 '12 at 15:13

If you have $m$ observations, all obtained independently and with equal probability from a set of $k$ possibilities, the probability that there are two identical observations is $1 - k(k-1)k-2)\cdot \dots \cdot (k-m+1)/k^m$ if $k \ge m$ and it's 1 if $k < m$. In your case, $k = 36500$ (100 years with 365 days each). For $m > 222$, this probability is $ > 1/2$.

In your particular problem, an out-of-bounds check on the year and day of the month might also be successful, to detect fake dates like 6/31/1960 or 7/2/2020.

| cite | improve this answer | |
  • $\begingroup$ I'm not sure if I follow, where does 222 come from...should that be a different letter? I guess it might help to say I have 10,000 records. What n do I suspect ? $\endgroup$ – Pureferret Nov 27 '12 at 15:19
  • 2
    $\begingroup$ The $222$ comes from $\sqrt {2 \ln 2 \cdot 36500}$. If you are interested, you could look at "The generalized birthday problem in Wikipedia $\endgroup$ – Ross Millikan Nov 27 '12 at 15:51
  • 1
    $\begingroup$ The condition m > 222 (not k > 222, that was a typo) comes from a calculation (see Ross Millikan's comment just above). With m = 10,000 records, each from m = 365000 possible choices, you are virtually certain to see at least one duplication. In fact, if you keep drawing records at random, the expected number of duplications of previous selections after $m$ draws is $m-k+k(1-1/k)^m$, which is about 1,200 for this choice of $m$ and $k$. See en.wikipedia.org/wiki/Birthday_problem . $\endgroup$ – Hans Engler Nov 27 '12 at 15:53
  • $\begingroup$ @Hans that last bit of info is actually really useful. $\endgroup$ – Pureferret Nov 27 '12 at 16:05
  • $\begingroup$ @ Pureferret - I thought so :) $\endgroup$ – Hans Engler Nov 27 '12 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.