Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I've analysed newspapers by counting the language distributions of the articles.

The results look like that:

Day 1               Day 2              Day 3

Economy             Economy            Economy
language 1: 0,35    language 1: 0,30   language 1: 0,90
language 2: 0,11    language 2: 0,10   language 2: 0,00
language 3: 0,54    language 3: 0,60   language 3: 0,10

Sports              Sports             Sports
language 1: 0,40    language 1: 0,30   language 1: 1.00
language 2: 0,20    language 2: 0,20   language 2: 0,00
language 3: 0,40    language 3: 0,50   language 3: 0,00

I've have already posted another question on that topic (Remove statistical outliers), but here comes my second problem. First of all, I want to remove all statistical outliers from data (e.g. day 3), to make it "clean" (see my other question (other post). After that, I want to terminate which changes in my data are just "noise" and witch are significant changes. But I'm not sure how to do it.

I was thinking of the following approach:

I could calculate the standard deviation (like in my other post) and treat every value outside of it as a "significant change". But I think this will cause a mistake if all my values are slightly increasing or decreasing.

Is there any mathematical technique to find the significant changes in my data?

Thanks in advance.

(PS: I have only a few samples (~ 150 days).)

share|improve this question

1 Answer 1

up vote 2 down vote accepted

Removing outliers does not necessarily "make the data clean". The outliers may belong . They could be a sign of a real phenomenon. What you are doing is like throwing the baby out with the bath water. Finding outliers to study them to determine if they are errors or something interesting is a different story. That is useful statistics. But statistics cannot tell you the data is bad just because it is an outlier. I have mentioned on the Cross Validated site that Dixon or Grubbs test can be used to find an outlier or modified Dixon to find two or more. Lets just for my sake suppose for the moment that your original data had no outliers, so nothing got deleted. Your question has no answer. To ask if an observation is pure noise versus a single with possible added noise cannot be tested statistically without an underlying model. For example if your series were a time series you could compare models. Is the series signal+noise or just noise. If the process is stationary you could test this using the autocorrelation function(testing that all lagged correlations are 0). But you are trying to do this sort of thing to isolated observations outside the context of a statistical model.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.