Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I've analysed newspapers by counting the language distributions of the articles.

The results look like that:

Day 1               Day 2              Day 3

Economy             Economy            Economy
language 1: 0,35    language 1: 0,30   language 1: 0,90
language 2: 0,11    language 2: 0,10   language 2: 0,00
language 3: 0,54    language 3: 0,60   language 3: 0,10

Sports              Sports             Sports
language 1: 0,40    language 1: 0,30   language 1: 1.00
language 2: 0,20    language 2: 0,20   language 2: 0,00
language 3: 0,40    language 3: 0,50   language 3: 0,00

So for instance on day 1, 35 % of the Economy-articles are written in language 1, 11 % in language 2 and so on.

Now I want to remove the outliers (like e.g. day 3) from my data. I was thinking about calculating the double standard deviation and remove all the values that are outside of it.

Does that make sense? Is there a problem if my values don't have a normal distribution? Or is there another way to get rid off the outliers?

In the end I want to calculate the average of all language 1, language 2 etc. values of each category over time and see how the values change.

Any technique how to do that?

Thanks in advance.

share|improve this question

1 Answer 1

up vote 1 down vote accepted

As I said in response eto your other question. Dixon's ratio test is a good way to detect a large outlier. What you are suggesting is called a 2-sigma rule. Two problems with it are (1) Approximately 5% of the population would be included in the group you throw out in addition to outliers the presumably came from a different distribution and (2) you don't know the variance, so you have to estimate it using all the data, but outliers inflate the variance thus helping to hide themselve. Dixon's test doesn't have these problems. Again I caution that identifying outliers does not necessarily tell you that you should remove them.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.