Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Say $d+1$ independent, equally reliable experts give you probabilities $p_0,\ldots,p_d$ of an event A occurring. What should you think the probability of $A$ is?

I'll give my solution (Edit: I retract my claim to having a solution. At the very least this is not, in general, normalized correctly. There are also other serious issues as pointed out in the comments and answers): $$ \begin{align} &P(A \mid p_0,\ldots,p_d) \\ &= \frac{P(p_0,\ldots,p_d \mid A) P(A)}{P(p_0,\ldots,p_d)}\\ &= P(A) \prod_{l=0}^{d}\frac{P(p_l \mid A)}{P(p_l)}\\ &= P(A) \prod_{l=0}^{d}\frac{P(A \mid p_l)}{P(A)}\\ &= \frac{1}{P(A)^d} \prod_{l=0}^{d} P(A \mid p_l)\\ &= \frac{1}{P(A)^d} \prod_{l=0}^{d} p_l\\ \end{align}$$

I think this is reasonable (let me know if you disagree) but I have a number of questions about it:

  1. I have couched this in an explicity Bayesian manner and I'm not sure if this is necessary. Can you instead talk about some estimates of $p_l$?

  2. Is line 3 ok? I'm dividing by zero... Can this be fixed by considering some density function?

  3. Why do we divide by $P(A)^d$ (and not to the power $d+1$ say)? It makes sense to me to multiply the probabilities together, I guess this is just normalizing?

  4. Does $P(A \mid p_l)=p_l$? It certainly seems like it should.

share|improve this question
2  
Could you justify $$P(p_0, p_1, \ldots, p_d) = \prod_{i=0}^{d} P(p_i)$$ a little? In many cases, conditionally independent events (e.g. the justification for $$P(p_0, p_1, \ldots, p_d|A) = \prod_{i=0}^{d} P(p_i|A)$$ are not necessarily independent in the absence of conditioning. –  Dilip Sarwate Mar 21 '12 at 20:35
    
I don't think your combined estimate is justifiable. If even a single expert predicts $0$ then their estimate overwhelms the effect of all the other estimates! –  Qiaochu Yuan Mar 21 '12 at 21:39
    
Thanks for your answers and comments everyone. I had a feeling this might be controversial and you haven't disappointed. I retract my claim to having a solution. –  Martin Leslie Mar 22 '12 at 0:05
add comment

2 Answers 2

About 2., we can think of a discrete case for a start and thus avoid the complications of conditional probabilities in the continuous case.

The main problem lies in 4. There's no basis for this assumption; you've killed the entire complexity of the problem by making that assumption. $P(A\mid p_l)$ depends on how reliable you think the experts are; $P(A\mid p_l)=p_l$ expresses that you believe that expert $l$ is always right. In the other extreme, if you think the "experts" are all completely useless, you'd have $P(A\mid p_l)=P(A)$. The conditional probability distribution you get with your assumption will generally not be normalized unless the experts are in fact always right.

share|improve this answer
add comment

To expand a little on the (very good) point that the user joriki gives in his answer, one potential problem is that you are conflating the event $E_{i} = (\textrm{Expert i was correct})$ with the event $G_{i} = (\textrm{Expert i gave estimate } p_{i})$. Only in the former case can you assume that $P(A|E_{i}) = p_{i}$. But just because expert $i$ said that the probability is $p_{i}$, it doesn't mean that's correct. Maybe you know that expert $i$ always forgets to carry the two when doing long division or something.

If we assume that the expert guess $p$ is a random variable with some 'expert' distribution $P(p)$, then under your assumption, another way to put it is like this: $$ P_{true}(A) = \sum_{i=0}^{d}p(A|p_{i})\cdot{}P(p_{i}) = \sum_{i=0}^{d}p_{i}\cdot{}P(p=p_{i}) = E_{\textrm{exp}}[p].$$

Basically, this means that your given simplifications imply that you have to make the assumption that the experts are an unbiased estimator of the true probability, that is, the expectation under the distribution over expert opinion equaling the true probability is a consequence of your derivation but it may not be true about independent experts out in the world.

The formula you give has weird (not necessarily bad) properties for certain test cases. For example, consider a case where your prior belief is $P(A)=1/2$ and there are two experts, (so $d=1$) who give probabilities $p_{0}$ and $p_{1} = \frac{n-1}{n}$, respectively, for some large value of $n$. Then

$$P(A|p_{0},p_{1}) = \frac{1}{1/2}\cdot{}p_{0}\cdot{}\frac{n-1}{n} \to 2\cdot{}p_{0}.$$ So, as expert 2 becomes more and more certain, the probability converges to twice the guess of expert 1. Clearly, this doesn't make sense for $p_{0} > 1/2$, and it might make even less sense subjectively if we value the estimates of the two experts equally.

I think part of the problem is assuming that $P(A|p_{i})=p_{i}$, because it means that if expert $i$ makes guess $p_{i}$, then all the other guess don't matter. But this can't be true for all of the $i$ experts simultaneously...

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.