Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Lets imagine that we have some experiments. Each experiment may result in one of the outcomes: A, B, C. So we have probability distributions for each experiment $P_A, P_B, P_C$ which is context-dependent, e.g.:

  1. $Context_1 \Rightarrow \{P_A^1, P_B^1, P_C^1\},$ experimental outcome is A
  2. $Context_2 \Rightarrow \{P_A^2, P_B^2, P_C^2\},$ experimental outcome is B
  3. $Context_3 \Rightarrow \{P_A^3, P_B^3, P_C^3\},$ experimental outcome is A
  4. $Context_4 \Rightarrow \{P_A^4, P_B^4, P_C^4\},$ experimental outcome is C

These probabilities are calculated by some function $F:Context\rightarrow \{P_A, P_B, P_C\}$

I want to estimate an absolute trust rate of this function. In other words, I want to be able to say "we can trust this function on 86%" like we do when we deal with Pearson's chi-square test.

Any suggestions?

share|improve this question
    
I'm confused by the wording "How can I verify my predictions now without any knowledge about the real distribution?", can you please edit the OP to explain in more detail, as clearly as possible, what you are trying to verify. I don't think you can ever "verify a prediction" without doing an actual experiment. Are your trying to predict the way your opponent will change their strategy in reaction to the way you change yours? –  Matt Calhoun Nov 25 '10 at 21:49
    
I'm not sure how relevant this will be for you, since it deals with thermodynamic systems in equilibrium; but you might be interested in L.D. Landau, Volume 5, 3rd ed, section 118, "Correlation of Fluctuations in Time". –  Matt Calhoun Nov 25 '10 at 21:55
    
Re-formulated the question, hope it is more clear now –  levanovd Nov 26 '10 at 8:26
    
Seems to me you need a Bayesian reformulation of your problem: $\mathbb{P}(X|Context)$. –  Raskolnikov Nov 28 '10 at 15:51

2 Answers 2

up vote 1 down vote accepted

If your predictions are repetitive I think you can do this. For example, if you always predict 50% chance of one choice, 30% for a second and 20% for the third (even if they are permuted on a particular play) you can do a standard chi-square test. Did you in fact get 50% right on the one you claim?

If your predictions vary continuously, you can still pick the highest probability and compute the observed number of times it is right compared to the prediction and the variance using the variance on each prediction is p(1-p). Then you can compare with the null hypothesis of 1/3, 1/3, 1/3

I'm not sure how to understand the statement "we can trust this function on 86%". I think the chi-square test can say "it is 86% probably that my predictions are better than the null hypothesis", but that doesn't tell you that the predictions are right.

share|improve this answer

Your function is really

$$F: Context \mapsto \mathbb{P}(X|Context)$$

or in other words you just want to determine the probability of a particular outcome, conditional on a certain context. The question is, what are your data? Do you know about the different contexts? Or do you only dispose of the outcomes?

Anyway, since $$\mathbb{P}(X|Context)=\frac{\mathbb{P}(X \cap Context)}{\mathbb{P}(Context)} \; ,$$ I think any test you'd want to devise for this function would boil down to the Chi-squared test anyway.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.