1
$\begingroup$

I have a sample of graphs (more than 10000...). that look like in the image below: enter image description here

I am searching for an unsupervised learning algorithms that can help me to detect anomalous observations.

Here what I suggest for beginning: for every observation I have a collection of points $(x,y)$. With this collection, I find Fourier series with regression (I compute coefficients with the base $\{1,\sin(x),\cos(x),\sin(2x),\cos(2x)\dots\}$). Now I have a set of coefficients instead of waves.

Somebody have an idea how to detect anomaly?

$\endgroup$
  • $\begingroup$ Too broad/ambitious $\endgroup$ – leonbloy Nov 26 '14 at 17:37
  • 1
    $\begingroup$ can you explain the figures? are they all normal obsevations or? In other words what do you call as anomalous observation? $\endgroup$ – Seyhmus Güngören Nov 29 '14 at 17:52
  • $\begingroup$ The figures are waves of analogue data (i.e volts of some machine). Anomalus observation are observations that very unsimilar (with specific algorithms that i look for) to the train data. $\endgroup$ – dmitriy Nov 30 '14 at 7:32
1
$\begingroup$

FFT (Fast Fourier Transform) each wave to get its Power spectral density (PSD).
Your "anomalous" waves may transform to a PSD like that of white noise, flat (however I'm no expert).

To refine this, make a Linear classifier. Train it manually: pick out anomalous 10 ? out of 100 ? waves by hand and eye, and make a linear classifier with e.g. scikit-learn to separate the 10 from the 90 / 10k from 90k.
Added 1 Dec: Spectral flatness is a measure of how noise-like a signal is; it's easy to implement, and may be adequate for your task.

(I'd suggest asking over on dsp.stackexchange.com, with tag Matlab or Python too.
Moderators, move the question ?)

$\endgroup$
  • $\begingroup$ I use sin/cos to transfer the waves to data that i can wotk with. $\endgroup$ – dmitriy Nov 30 '14 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.