I have measured the frequency of terms within the same period (say 14 days), and I wish to use Standard Deviation to calculate their burstiness (i.e. which terms are more spiky), therefore, if a term is constantly high count but due to their low spike, the burstiness hould be still low.

I used Standard Deviation (SD) to calculate but the score came up quite high, if I normalised the frequency using the max frequency then the significance is not big. Therefore, my questions are:

(i) Is SD a correct method to use? Or there are other better ways? (ii) Is there any function to make the score more meaningful and understandable?


As I understand it, the data have terms in one column and their frequencies(in 14 days) in another column. You want to know which terms have more frequency within a period of 14 days.

There are plenty of methods you can use. You have not described how you used standard deviation to find which of the terms have high frequency and which of them have low frequency, so I can't tell whether your method is tight or not- please give some details in order to understand it better.

Anyways, the easiest method would be to construct a confidence interval, which would of course depend on data. The main idea is to estimate the mean frequency of terms from the data and check whether a given observation differs significantly from the mean.

Let $X_1,X_2,\cdots X_n$ be the frequencies of n terms. Then mean is given by $\displaystyle\mu=\frac{1}{n}\sum_{i=1}^n X_i$. Then, for each $X_i$ for $1\le i \le n$, we can have a look at the difference $(X_i-\mu)$. Let $\sigma$ be the standard deviation of the data. Usually, a confidence interval of $\pm 3\sigma$ from $\mu$ is taken, i.e. from $\mu-3\sigma$ to $\mu+3\sigma$.

If $|X_i-\mu|>3\sigma$, you can say that $i^{th}$ term is spiky.

Why $3\sigma$ and not $\sigma$ or $2\sigma$? that depends upon various things, such as power of your test. For a general discussion on confidence intervals, see this link.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.