Chi-square analog for context-dependent distributions

Lets imagine that we have some experiments. Each experiment may result in one of the outcomes: A, B, C. So we have probability distributions for each experiment $P_A, P_B, P_C$ which is context-dependent, e.g.:

1. $Context_1 \Rightarrow \{P_A^1, P_B^1, P_C^1\},$ experimental outcome is A
2. $Context_2 \Rightarrow \{P_A^2, P_B^2, P_C^2\},$ experimental outcome is B
3. $Context_3 \Rightarrow \{P_A^3, P_B^3, P_C^3\},$ experimental outcome is A
4. $Context_4 \Rightarrow \{P_A^4, P_B^4, P_C^4\},$ experimental outcome is C

These probabilities are calculated by some function $F:Context\rightarrow \{P_A, P_B, P_C\}$

I want to estimate an absolute trust rate of this function. In other words, I want to be able to say "we can trust this function on 86%" like we do when we deal with Pearson's chi-square test.

Any suggestions?

-
I'm confused by the wording "How can I verify my predictions now without any knowledge about the real distribution?", can you please edit the OP to explain in more detail, as clearly as possible, what you are trying to verify. I don't think you can ever "verify a prediction" without doing an actual experiment. Are your trying to predict the way your opponent will change their strategy in reaction to the way you change yours? –  Matt Calhoun Nov 25 '10 at 21:49
I'm not sure how relevant this will be for you, since it deals with thermodynamic systems in equilibrium; but you might be interested in L.D. Landau, Volume 5, 3rd ed, section 118, "Correlation of Fluctuations in Time". –  Matt Calhoun Nov 25 '10 at 21:55
Re-formulated the question, hope it is more clear now –  levanovd Nov 26 '10 at 8:26
Seems to me you need a Bayesian reformulation of your problem: $\mathbb{P}(X|Context)$. –  Raskolnikov Nov 28 '10 at 15:51

If your predictions are repetitive I think you can do this. For example, if you always predict 50% chance of one choice, 30% for a second and 20% for the third (even if they are permuted on a particular play) you can do a standard chi-square test. Did you in fact get 50% right on the one you claim?

If your predictions vary continuously, you can still pick the highest probability and compute the observed number of times it is right compared to the prediction and the variance using the variance on each prediction is p(1-p). Then you can compare with the null hypothesis of 1/3, 1/3, 1/3

I'm not sure how to understand the statement "we can trust this function on 86%". I think the chi-square test can say "it is 86% probably that my predictions are better than the null hypothesis", but that doesn't tell you that the predictions are right.

-

$$F: Context \mapsto \mathbb{P}(X|Context)$$
Anyway, since $$\mathbb{P}(X|Context)=\frac{\mathbb{P}(X \cap Context)}{\mathbb{P}(Context)} \; ,$$ I think any test you'd want to devise for this function would boil down to the Chi-squared test anyway.