1
$\begingroup$

INTRODUCTION:

I recently had a new idea on how to measure the dispersion of data. Since I have done my quick research and have not been able to find this method anywhere I was curious to see what somebody with greater expertise in the field of statistics would think about it and so I am sharing it here as a first step.

During my engineering education I have been taught several ways to analyze the dispersion of data, the most popular being the standard deviation, where a lot of the probability and statistics theory is based on. In some other applications however, I have been exposed to methods such as the range, both of the MAD's or even the IQR. I even did my own literature study regarding the suitability of each of these methods, and it is clear to me that while the standard deviation is the most widely used method, it is not the only method nor always the most suitable one.

IDEA:

To my idea: The result of this method actually looks a lot like the box and whiskers plot you would get from using the IQR (Inter quartile range) method. Instead of relying though on percentiles, it relies on averages.

Here the method outline:

  1. Calculate the mean: This is a very common step, since most methods out there require this value in order to be able to calculate the dispersion of the data from the expected value.

  2. Calculate the mean for left and right sides: We just calculated the mean, which for a function represents the centroid of a distribution. This center line is used usually as a reference for how far away a data point is in terms of dispersion. So why do we not split the data here, and take again the averages, but this time only for each half, left and right? That is actually what I then do.

    1. Results: We now have a mid-average (mq), a lower quarter average (lq) and a upper quarter average (uq). Obviously from this we can then also calculate the IQR as: \begin{equation} \text{IQR} = \text{uq} - \text{lq} \end{equation} Obviously this gives a good idea on how the data is distributed. Not only can we say something about its distribution through the IQR, but we can analyze the symmetry by checking the difference between the $(\text{uq} - \text{mq})$ and $(\text{mq} - \text{lq})$.

I hope I am also right in saying that the IQR (for a normal distribution) represents 50% of the data, and if we would want to use more data, we would simply have to repeat the 'split and calculate new average' method (in which the next iteration would include 75% of the data).

So, what do you guys think about this method? Maybe you could also tell how one would go about testing the method?

OWN THOUGHTS:

As far as I can see, the method relies on averages, which is quite nice. Unlike the percentiles used for the typical IQR method, averages rely on the whole data. This might not make them very robust, but are always adjusted fully to the data.

In addition, I find this quite an intuitive method; it's simple and the dispersion measured by 50% of the data seems easy to grasp.

Finally, compared to the standard deviation, the values do not need to be squared, which means that this method does not get skewed as much by outliers and may be more fitting when analyzing data with errors. (The reason the mean absolute deviation (MAD or MD) is sometimes used.)

EXAMPLE:

In order to give an idea how this would look like, two examples, one in which the numerical process is shown, another of a comparison between the box and whiskers (without the whiskers) results from the typical IQR method and the method I propose here.

Example 1: Say we start off with the following data:

2,4,4,4,5,6,6,7,8,10

From this we can calculate the mid-average: \begin{equation} \text{mq} = \frac{2+4+\cdots+10}{10} = 5.6 \end{equation} All the values lower than or equal to 5.6 are taken and the lq is calculated with the same average formula: \begin{equation} \text{lq} = \frac{2+4+\cdots+5}{5} = 3.8 \end{equation} We can repeat this method to calculate the uq, this time for all the values bigger or equal to 5.6: \begin{equation} \text{lq} = \frac{6+6+\cdots+10}{5} = 7.4 \end{equation} We therefore can calculate the IQR using these average values: \begin{equation} \text{IQR} = \text{uq} - \text{lq} = 3.6 \end{equation} In addition, we can observe that when taking the ratio between $(\text{uq} - \text{mq})$ and $(\text{mq} - \text{lq})$, we get a value of 1.0, suggesting that the data is symmetric.

Example 2: Here the figure I have obtained using a Python code (posted below) which compares the typical IQR method with the method outlined here. In green the mq, lq and uq for the typical IQR method, in red the result of the method I have been proposing. You can see in green the mq, lq and uq for the typical IQR method, in red the result of the method I have been proposing.

The sample size is 1000, and the values for the lines are:

My method:

  • mid_avg: -0.0905134149804
  • lq_avg: -1.63815314319
  • uq_avg: 1.50107551444
  • IQRwAVG: 3.13922865764

Typical IQR method:

  • median: -0.116056069599
  • 25 percentile: -1.39684011872
  • 50 percentile: 1.21390120378
  • IQR: 2.6107413225

Here the code:

# -*- coding: utf-8 -*-
"""
Created on Sat Jan 13 20:47:36 2018

@author: Marco DS

This script shall give an idea how well the IQR 
method works if the quartiles are calculated using
averages instead.

In this version we will focus on comparing the IQR
with quarter averages to the typical IQR method 
using percentiles.
"""

import numpy as np
import matplotlib.pyplot as plt

#==============================================================================
# Create random sample
#==============================================================================
mu, sigma = 0.0, 2.0
np.random.seed(0)
sample = np.random.normal(mu, sigma, 1000)

#==============================================================================
# Calculations
#==============================================================================

#------------------------------------------------------------------------------
# IQR with averages
#------------------------------------------------------------------------------
mid_avg = np.average(sample)

lq_avg = np.average(sample[sample<=mid_avg])
uq_avg = np.average(sample[sample>=mid_avg])

IQRwAVG = uq_avg - lq_avg

print '=========='
print 'mid_avg: ', mid_avg
print 'lq_avg: ', lq_avg
print 'uq_avg: ', uq_avg
print 'IQRwAVG: ', IQRwAVG

#------------------------------------------------------------------------------
# IQR with averages
#------------------------------------------------------------------------------
p75, p50, p25 = np.percentile(sample, [75 , 50, 25])
IQR = p75 - p25

print '=========='
print 'median: ', p50
print '25 percentile: ', p25
print '50 percentile: ', p75
print 'IQR: ', IQR

#==============================================================================
# Show plot
#==============================================================================
plt.hist(sample, 10)

#plt.xlabel('Smarts')
#plt.ylabel('Probability')
#plt.title('Histogram of IQ')
#plt.text(60, .025, r'$\mu=100,\ \sigma=15$')
#plt.axis([40, 160, 0, 0.03])
#plt.grid(True)

for i in [lq_avg, mid_avg, uq_avg]:
    plt.axvline(x = i, color = 'r')

for i in [p75, p50, p25]:
    plt.axvline(x = i, color = 'g')

plt.show()
$\endgroup$
  • $\begingroup$ "I hope I am also right in saying that the IQR (for a normal distribution) represents 50% of the data" I'm afraid there is no reason to believe this is true. Indeed, in both your examples your "IQR" contains well more than 50% of the data (in the first example it contains 7 out of 10 points; in the second it is wider than the standard IQR which by definition contains 50% of the data). $\endgroup$ – Rahul Jan 16 '18 at 2:56
  • $\begingroup$ Have you actually compared your idea to the standard deviation on data that is highly skewed or has outliers? I expect it will be just as bad. $\endgroup$ – Rahul Jan 16 '18 at 2:58
  • $\begingroup$ Rahul, I have actually not yet tried that, but its not a bad idea. I will go ahead and try that later. $\endgroup$ – user3604362 Jan 16 '18 at 9:55
  • $\begingroup$ Here's a simple one: 1, 1, 2, 2, 2, 2, 3, 4, 5, 10. The quartiles are 2, 2, 4, so the IQR is 2. Your method gives a much bigger range contaminated by the outlier. $\endgroup$ – Rahul Jan 16 '18 at 16:44
0
$\begingroup$

What is "most suitable" depends on the context and purpose.

Suppose $$\tag 1 X_1,\ldots,X_n\sim N(\mu,\sigma^2).$$

Suppose we take the upper average minus the lower average of this sample of size $n.$ That is a random variable whose expected value is $c\sigma,$ where $c$ is a number that can be computed without knowing $\mu$ or $\sigma.$ Since we don't need to know $\mu$ or $\sigma$ in order to know $c,$ we can use $(\text{upper average} - \text{lower average})/c$ as an unbiased estimator of $\sigma,$ and if smallest mean squared error or some other criterion is insisted on, we might use a different number from $c.$ Now suppose it were shown (if I'm not mistaken, it has been) that a specified multiple of the sample standard deviation performs better as an estimator of $\sigma$ than does $(\text{upper average} - \text{lower average})/c,$ in the sense of being an estimator with a smaller variance. Then you're better off with the sample standard deviation.

However, the population from which you sample may not be normally distributed, and perhaps has long tails. In that case, your estimator may be better.

But what if we consider the upper average minus the lower average for the population rather than for the sample? That would be $\alpha=a\sigma$ for some number $a$ that can be computed without knowing $\mu$ or $\sigma^2.$ Then the pair $(\mu,\alpha)$ can work just as well as a means of parametrizing the family of normal distributions as can $(\mu,\sigma).$ But variances have this advantage (which is why they're widely used):

Suppose $X\sim N(\mu,\sigma^2)$ and $Y\sim N(\nu, \tau^2)$ and $X,Y$ are independent. Then $\operatorname{var}(X+Y) = \sigma^2 + \tau^2.$

Your proposed upper average minus lower average may have some good uses, but so do the more standard measures of dispersion.

$\endgroup$
  • $\begingroup$ Hey Michael, unfortunately I did not understand quite well what you mean with “...where c can be calculated without knowing mu or sigma.” As far as I can tell from your explanation, the standard deviation arises naturally when dealing with the math, which is why when working with multiple random variables, they stack up quite nicely. $\endgroup$ – user3604362 Jan 16 '18 at 9:53
  • $\begingroup$ Just a thought I had: Are you trying to explain a method to compare the efficiency of the estimator? If yes, I did not understand your method, but I must admit I have not yet done the calculations. I will do so asap and update this page accordingly. $\endgroup$ – user3604362 Jan 16 '18 at 10:01
  • $\begingroup$ @user3604362 : One thing you've got backwards: The fact that variances are used a lot is not why they "stack up quite nicely"; rather the fact that they "stack up quite nicely" is why they're used a lot. $\endgroup$ – Michael Hardy Jan 16 '18 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.