Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Can anyone give a definition on empirical accuracy?

I Googled this keyword but cannot find a satisfactory definition. It seems to be the accuracy rate of a specific sample.

Any link or typed text will be appreciated.

I saw it in an data mining book.

The context is evaluating the accuracy of decision tree by inputing the model test data.

The original test is like this:

Given the test set that contains N records, let X be the number of records predicted by a model and p be the true accuracy of the model.

By modeling the prediction task as a binomial experiment, X has a binomial distribution with mean Np and variance Np(1-p).

And the empirical accuracy, acc=X/N, also has a binomial distribution with mean p and variance p(1-p)/N.

share|improve this question
4  
The usual question: where did you see this term being used? –  J. M. Sep 18 '11 at 14:33
    
thanks for the recommendation, I made some edit. @J. M –  xiaohan2012 Sep 18 '11 at 15:22
    
Cool, a data mining book. Which one? –  J. M. Sep 18 '11 at 15:25
    
@J. M, Introduction to Data Mining, by Pang-Ning Tan, Michael Steinbach and Vipin Kumar –  xiaohan2012 Sep 18 '11 at 15:39

1 Answer 1

up vote 0 down vote accepted

Following the definition given in your question, the empirical accuracy is the proportion of records correctly predicted by the model.

For example, if you tested your model on 100 records and it correctly predicts 70 of them, then the empirical accuracy is 70 / 100 = 0.7, or 70%.

If the model has a true accuracy of $p$ (i.e. in the long run it correctly predicts a proportion $p$ of all records) then there is a probability $p$ that any individual record will be correctly predicted. You can think of the prediction of record $i$ as a random variable $X_i$ which takes the value 1 with probability $p$, and 0 with probability $1-p$.

If you are testing $N$ records, then each one individually has probability $p$ of being correctly predicted, then the total number of records correctly predicted is

$$X = \sum_{i=1}^N X_i$$

which is itself a random variable. Being the sum of $N$ identically distributed Bernoulli random variables, the distribution is known to be Binomial(N,p). This distribution has mean $Np$ and variance $Np(1-p)$, as the book describes.

The empirical accuracy is the proportion of records predicted correctly, i.e. it is given by

$${\rm acc} = X/N$$

Here $N$ is just a constant, and we use the following rules which apply to any random variable $X$ (subject to a couple of technical conditions you don't need to worry about):

  • ${\rm E}(cX) = c{\rm E}(X)$
  • ${\rm Var}(cX) = c^2{\rm Var}(X)$

Here ${\rm E}(X)$ is the expectation or mean of $X$, and ${\rm Var}(X)$ is the variance of $X$.

We have multiplied $X$ by $1/N$ to get acc, so the mean of acc is $$(1/N) \times Np = p$$ and the variance of acc is $$(1/N)^2 \times Np(1-p) = p(1-p)/N$$ as claimed in your book.

share|improve this answer
    
Really impressive! Thanks –  xiaohan2012 Sep 19 '11 at 11:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.