I have a dataset like so (left column produced by an estimation algorithm, right column being what happened in reality). [EDIT: Please note that every event is distinct (i.e., we are not repeatedly testing the same event, but every time we are making a prediction about some new kind of thing that may or may not happen).]

event probability        actual outcome (whether event occurred)
0.939658077              TRUE
0.705453465              FALSE
0.310251296              TRUE
0.385363009              FALSE
0.660532932              FALSE
0.290306978              TRUE
0.484473665              FALSE
0.01615261               FALSE
0.898152645              TRUE
0.389938993              TRUE
0.032598374              FALSE
0.599836035              FALSE
0.428701779              TRUE
0.7787285                TRUE
0.14356366               FALSE
0.65105148               FALSE
0.418174021              FALSE
0.724846388              TRUE
0.844266775              TRUE
0.437018647              TRUE
...                      ...

How can I evaluate the quality of the prediction algorithm? (Assume data set size is large enough.)


EDIT: So, for example, if the estimated probability is 0.5, the model is saying it doesn't know what to predict, so in a way there is 0 error whatever the outcome. And the model could estimate a 0.9 probability of the event occurring, and once in ten times you would still expect it to not occur. However, over the full dataset, if the model keeps saying 0.1 and then the event usually occurs, and if it keeps saying 0.9 and the event usually does not occur, then it's performing poorly.


Confusion Matrix and ROC curves probably will suit you.

Check this out: https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5

  • $\begingroup$ Could you explain more instead of just giving a link? $\endgroup$ – Larry Jan 10 '19 at 14:48
  • $\begingroup$ thanks for the link, I'm reading up on your suggestion right now! (but why is the answer already appearing as accepted, before I even look at it? i tried to un-accept it just to see if I can control the feedback [not because I have any problem with the answer - I've just started reading the linked page] but nothing happened) $\endgroup$ – sesquipedalias Jan 10 '19 at 16:56
  • $\begingroup$ hmmm, the suggested solution requires classification into predefined classes, but as I put in the title of the question, here we have no classes: we just have a probability of an event occurring, and then the event may or may not occur (of course, either result is consistent with any estimation other that 0 or 1)... and we need to somehow measure how good the estimations of the event probability are, from the entire dataset of predictions vs outcomes... thanks... (I'll think about the linked content some more, though...) $\endgroup$ – sesquipedalias Jan 10 '19 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.