0
$\begingroup$

I have a numerical dataset in which the standard deviation is larger than the mean and I am interested to know what this fact can tell me about the dataset. I have drawn the following conclusions on my own:

1. The mean is skewed by negative values, extreme outliers or both
2. The dataset is not normally distributed
3. The median will provide a better estimate of a "typical" value than the mean

Can you find any flaw in them? Are there any other conclusions I can draw about this dataset from the fact that the standard deviation is larger than the mean? I'm interested in:

- implications for describing such a dataset, 
- deciding how "typical" a given value or subset of values from the dataset are, 
- performing analyses on the dataset as a whole 

Sorry if this is overly-vague/theoretical, I have many datasets that I can supply as examples along with specific questions I have about each but I'm choosing to ask a more general question in hopes of getting a deeper understanding about what this phenomena means.

$\endgroup$
0
$\begingroup$

(1) is meaningless because "and" and "or" do not have the same meaning.

(1) using "or" is trivially true because any normal distribution has negative values.

(1) using "and" is false because a distribution of a non-negative random variable might have a mean but infinite standard deviation, or because a normal distribution can have negative mean.

(2) is typically true, because the median is the 50th percentile, and is robust, whereas the mean can be changed arbitrarily by even a single data point, and thus often gives a misleading picture when the data is not normally distributed (like income).

[Okay now with the new list...]

(2) is still false because any normal distribution with negative mean will have a (positive) standard deviation which would be greater than the mean.

In general, comparing the mean with the standard deviation is typically not meaningful.

$\endgroup$
  • $\begingroup$ Thank you. I will modify my conclusions list. I didn't realize how sloppy it was $\endgroup$ – Slavatron Jul 21 '14 at 16:37
  • $\begingroup$ @Slavatron: Perhaps you should specify what your data set is, otherwise it is difficult to imagine a scenario in which comparing the mean and standard deviation is meaningful. $\endgroup$ – user21820 Jul 21 '14 at 16:52
  • $\begingroup$ Today, I've seen this trend in the following datasets: a list of platinum-free intervals for cancer patients, a list indicating the percentage of non-cancerous cells found in removed-tumors, a list of the number of days a person survived after receiving a cancer diagnosis. In each of these three cases, there are either negative values in the dataset or extreme outliers. In fact, just about every time the standard deviation is larger than the mean I take it as a signal to examine the data more closely but I'm specifically interested in generalizations I can make or even expectations I can have. $\endgroup$ – Slavatron Jul 21 '14 at 17:03
  • 1
    $\begingroup$ @Slavatron: In that case I don't think you should compare the mean and standard deviation. I don't see why any of your data should be negative, so that should be separately filtered out directly by checking for negative values. Also if you want to detect outliers, you should check whether the data has a high kurtosis or (median-mean)/(stddev) is high. That would be far more robust in testing for the presence of outliers. $\endgroup$ – user21820 Jul 22 '14 at 3:53
  • $\begingroup$ @Slavatron: Sorry I meant "(median-mean)/(stddev) is large in absolute value". $\endgroup$ – user21820 Jul 22 '14 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.