Here's an initial sketch of what I would do:
People are too lazy for this. So, do it in reverse. Ahhhmm.
In a Web application ("Are you smarter ?"), give users two abilities:
- Take the test. (Largest weight on information gathered this way).
- Browse through tests other "users" took, and show only the questions, the score/grade, and maybe the time it took to finish.
With this, let the users (You choose one or more of these, and think of more):
- Rank the "smartness" (significant weight, with the option to disregard at a later time)
- Assert that they are smarter, i.e these are not so good test results (Minor weight, with the option to disregard at a later time). This would direct them to the beginning of a "similar" test (Large weight on this).
- Give the explicit option to take "this" test ("Similar") (Large weight on this).
In actual tests you should gather as much information as you can, i.e, etc..
This is a feedback loop, initialized with fabricated users, tests, and results.
Gradually/Slowly, it would/should:
1. Be able to generate better tests (Better means many things here), e.g like choosing the best book to learn a given subject.
2. Make "Similar" test really more and more similar.
3. Make the statistics closer and closer to the objective unknown you are actually after, either by real empirical data, a clever summary of subjective data, or (Most probably) a clever combination of these...