Proper way to calculate the mean of a dataset - use the entire dataset or the frequency table values?

Question

I'm not trained in mathematics, but I took a course about using statistics in understanding hotel data. One step of the process requires us to calculate the mean rate of a collection of stay records. However, the instructor first calculated a frequency table of all unique rates, and then determined the mean. He then determines the standard deviation and uses these two values to eliminate outliers.

MY question is whether it is correct to do it this way, or should the mean be calculated from the entire dataset? For example, the real world dataset I'm trying to apply this against actually has an inordinately large number of values at 299 and 329. So I if calculate the mean from just a list of the unique values without considering the frequency, I get a much lower mean than if I do it from the total set.

From my VERY basic understanding of statistics, it could be that the example dataset was normally distributed while my real-world one is quite negatively skewed, so maybe that makes the difference?

Any guidance or direction to resource material would be great. Thank-you, AF

Answer 1

Consider this small example where the dataset consists of just these numbers:

$$ 20, 40, 40, 100, 100, 100, 100, 100. $$

There are multiple ways to compute the mean. One is

$$ \frac{20 + 40 + 40 + 100 + 100 + 100 + 100 + 100}{8} = \frac{600}{8} = 75. $$

Another way is to look at the unique values and their frequencies: $$ \begin{array}{c|c} \text{value} & \text{frequency} \\ \hline 20 & 1 \\ 40 & 2 \\ 100 & 5 \end{array} $$

Then take the mean this way:

$$ \frac{20 \times 1 + 40 \times 2 + 100 \times 5}{1 + 2 + 5} = \frac{600}{8} = 75. $$

This works because

$$ 20 \times 1 + 40 \times 2 + 100 \times 5 = 20 + 40 + 40 + 100 + 100 + 100 + 100 + 100, $$

and because when you add up the frequencies of all the unique values, you get the total number of (not necessarily unique) values in the dataset. Notice that in both cases the numerator came out to $600$ and the denominator to $8$; that is not a coincidence.

If you have many more data values but only a small number of unique values among them, the second way is much easier to write out by hand, because it is much easier to work out $329 \times 1088$ (for example) than to write out $1088$ copies of the number $329$ with $+$ signs between them.

You are correct, however, that looking only at unique values and ignoring their frequencies will give you the wrong answer:

$$ \frac{20 + 40 + 100}{3} = \frac{160}{3} \approx 53.333. $$

With data that are symmetric and approximately normally distributed, you might get an accurate mean this way, because the errors will cancel out, but you would still get an incorrect standard deviation.