Confidence level in answer being correct when multiple people give that same answer

We are very rusty on our math, probability, statistics, permutations and combinations and need some help figuring out how to calculate some relative "confidence level" based on the number of people who give the same answer to a question. This is somewhat related to Wisdom of Crowds where if we ask a group of people the same question and many choose the same answer it is likely that answer is correct.

Answer to a question is either "a" or "b". If we ask 1 person we would have 50% confidence that is the right answer. If we asked two people and they both said the same answer we would be 75% confident it is right I think??

If we keep extending this:

Possible Answers = 4, we ask 2 people and they give the same answer ... it should be a higher confidence level but what? (87.5%?)

The more possible answers for a question and the more people who give the same answer should lead to a higher confidence level - but what is the formula or algorithm to calculate it?

-
– Charles Apr 28 '11 at 13:59

No, the 50% is already totally arbitrary. If you do not know anything about a probability distribution it does not imply equidistribution.

If you actually assume that people give random answers then there is no information at all in the fact that three people give the same answer.

-
What if we assume that everyone has a probability $p > .5$ of being right? – Joshua Shane Liberman Apr 28 '11 at 13:59
Oh, I agree that a meaningful question can be formulated in this context, but building the model comes first and after watching friends learn statistics for their economics degrees, I think that it is dangerous to give people formulas before they can understand their models. They will plug in stuff. The Wisdom of Crowds is based on the model that there is the same chance to guess too high than to guess too low which may be reasonable for guessing the sweets in a jar. – Phira Apr 28 '11 at 15:09

Refer to the computation of a binomial confidence interval. http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

Your "P" in the formula would be .5, which is the probability of being correct, by chance, given choices "a" or "b". "n" would be the number of people surveyed and "z" is the z-score. Once you obtain this confidence interval, subtract this result from 1.

The normal approximation formula is given as:

$$\hat{p}\pm z_{1-\alpha/2}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

-