0
$\begingroup$

I have similarity distribution graph, the graph has cosine similarity distribution values on y axis and relevant dates on x axis.(Datas are coming from a csv file) According to this values i need to find plateu to measure is there any high tension or not (something like is there any bump or not) my graph is as follows ;

The output of graphs is like myoutoutput1. I just calculated max value of graph and min value and average and i drawed a line for each ,also if number of values above the average greater than the number of values below the average then i said it to be a it is a high tension (i mean eventy topic)

what kind of math function should be used to measure that yes there is an event in your graph or not ? may be taking integral or etc.

One of sample for my csv file is like ;

date,close,topic
11-Jul-16,0.3919340235688502,T0
13-Jul-16,0.34676875219073117,T1
17-Jul-16,0.7318672747426419,T1
19-Jul-16,0.551091886884678,T1
21-Jul-16,0.4790156920874872,T4
23-Jul-16,0.49307051752070574,T0
25-Jul-16,0.28283343212895085,T0
27-Jul-16,0.31970262826528034,T0
29-Jul-16,0.4017937576119373,T4
31-Jul-16,0.28103157876820484,T3

for example when value goes from 0.3 to 0.7 there is a valuable increase and it is a tension but 0.3 to 0.5 not an eventy increase.

I mean that if distribution value goes like 5,7,8,10,12,9,8,10 there is no any eventy topic but if it goes like 5,7,8,18,19,9,8,7,5 it is an eventy one.

Actually the following article did what i want but i have trouble with understanding of eventy analysis, (Article:Using Topic Modeling and Similarity Thresholds to Detect Events)

Click for article

Thanks for your interest and sincerity..

$\endgroup$
  • $\begingroup$ What don't you understand about the paper's method for bump detection? $\endgroup$ – user3658307 Aug 24 '17 at 17:53
  • $\begingroup$ I apologize , I could not understand $\endgroup$ – mstfky Aug 24 '17 at 19:24
0
$\begingroup$

Let me try to describe the algorithm described in the paper. Let me know if it makes sense.

The authors have an algorithm for "Bump detection", which they run on several differently smoothed versions of their data (cosine similarity curves).

The basic idea is two define two horizontal lines on the plot of the data. The first line, the cold level $L_C$, defines the base of the bump, so all values except those of the bump are below $L_C$. The second line, the hot level $L_H$ defines the base of the plateau at the very top of the bump. Notice that $L_C$ and $L_H$ are defined by the data. In other words, given a plot, you can extract them via a simple algorithm that simply assigns the lines values from the data and tests if they define a bump (e.g. the data only crosses $L_C$ twice).

See the paper's figure 5 & 6 to see these visually.

A parameter is then chosen by the authors, which is the threshold $T$ for bump detection. The idea is that $L_C$ defines the bottom of the bump and $L_H$ the top, hence $L_H - L_C$ is directly related to the height of the bump; thus, we check if $$ L_H - L_C > T $$ If so, then a bump is considered to exist. I.e. if the bump is large enough (higher than $T$), it is considered "eventy".


Separately, here's one idea you might consider. Trying fitting a Gaussian function to the data and if (1) the variance $V$ is smaller than some threshold $T_V$ and (2) the error $E$ in the fit is smaller than some threshold $T_E$. If so, then there is a bump at the mean of the function.

The first criteria ensures that, if there is a bump, it is sufficiently thin (not a flat smear), while the second ensures that the fit is reasonable at all, i.e. using a bump function is at least moderately a good description of the data.

If you try it, let me know if it works.

$\endgroup$
  • $\begingroup$ Thanks for your sincerity and interest. In first opportunity i dont know that how to measure LC and LH. Should i take difference of each ordered array element one by one to compare difference with threshold ? Actually I will more focuse on fitting gaussian function to my array i mean your second offer. Then i will inform you about situation. I should make some read about Gaussian and how to apply it in to 1d array with above values. Do you have any idea about it ? about applying gaussian? Thank you very much again.@user3658307 $\endgroup$ – mstfky Sep 5 '17 at 7:53
  • $\begingroup$ I dealed with javascript to measure peaks in an array, after finding peak values i will compare them with defined threshold value. what should i use for threshold, what about difference of max and min value in my array ? then i defined a differencedValue to take difference of peak points value with my array average to compare this difference with threshold ? i m a bit more confused. forgive me.@user3658307 $\endgroup$ – mstfky Sep 5 '17 at 18:53
  • $\begingroup$ @mstfky If I remember correctly (might not be so), the paper uses a simple algorithm to get $L_C$ and $L_H$, basically just iteratively testing different values until there are only two intersections (the former starts low and moves up, the latter starts high and moves down, I think). Yes, you can use the change in sign of the difference with the threshold to determine when there are only two intersections, I believe. $\endgroup$ – user3658307 Sep 6 '17 at 2:35
  • $\begingroup$ @mstfky My apologies, but I don't know what your second comment is asking. Maybe you could post a new question with more details and your JS code on stack overflow and ping me there? $\endgroup$ – user3658307 Sep 6 '17 at 2:40
  • $\begingroup$ Thank you again , if you check above datas I mean distribution values , can you tell me what should be LC and LH for that values and also threshold ? I still could not introduce this simple algorithm :( $\endgroup$ – mstfky Sep 6 '17 at 2:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.