0
$\begingroup$

I need a statistical formula to capture a particular phenomena that I need to model in software.

I have a light that can be on or off. When turned on, it can be one of many colors (for example, let's say 10,000 possible colors).

I observe the light for a period of time (let's say 3 days) and want to know the degree of correlation between any two colors Cx and Cy. In other words, if Cx and Cy occur near each other in time more often than we would expect given independence and chance, I expect the formula to return a value approaching 1.

I'd like to understand if it's possible to have a correlation formula which captures the general phenomena, but there are some simplifying assumptions that could be made if the general case is too difficult:

1) We can assume when a color turns on that it always stays in that state for a fixed period of time (e.g. 10 seconds)

2) Each light color could be assigned a probability of turning on. That probability may change over time but is fairly stable.

3) I'm ok to set an assumed influence threshold. In other words, we know something about how colors affect one another and know that the one color can not influence another that occurs more than say 30 minutes in the future.

If there are other simplifying assumptions that would help, please raise them so we can discuss whether they apply.

I'm looking for both the correlation formula itself as well as help in understanding the intuition behind it. I have attempted to sole this problem in a few different ways, but they don't seem quite right. Here's what I have tried using the above assumptions #1-#3:

A) Co-Occurrence: count the number of times Cx and Cy co-occur in a 30min period (call it #Cxy) and divide that by (#Cx + #Cy - #Cxy). My issue with this approach is that it doesn't account for how likely co-occurence is by chance.

B) Phi Coefficient: http://en.wikipedia.org/wiki/Phi_coefficient. I had trouble reasoning through how to set the totals… it doesn't seem to map quite right.

C) I have considered viewing it as correlated poisson, but this doesn't seem a perfect fit either. E.g., Correlated Poisson Distribution

$\endgroup$

1 Answer 1

0
$\begingroup$

You didn't say explicitly, but I'm assuming there's only one light on at a time, that the light always on (just switching colors), and that there is no "off" state. If those assumptions are inappropriate, you'll only need light modifications to the following:

To determine if "Cx and Cy occur near each other in time more often than we would expect given independence and chance", you need some model of what chance occurrences would look like. This is your "null hypothesis": Are all colors expected to have equal frequency and for there them to be uncorrelated?

Once you've picked a null hypothesis, calculate Cxy for that (Expected) case. You can then calculate Cxy (your solution A) for the actual Observed data. You can then compare the Observed and Expected ratios using the G-test or chi-squared test, either of which will tell you whether there's statistically-significant deviation from your null hypothesis.

You can also compare changes between, say, the first and last day of testing to see if the correlations have drifted over that time.

As a further refinement, you might want to play with the period within which Cx and Cy are counted as co-occuring. You can either require it to be an immediate change, or lengthen the period, or do something more creative. (See below.) Varying your criterion for co-occurence may tell you interesting things about the system like "Blue often occurs right next to green, but over the course of a few hours, an increase in the frequency of green does not imply increase in the frequency of blue. Instead, green seems to move blue's occurrence closer to itself, but does not cause blue to occur if it would not have happened anyway."

"Creative" criteria for co-occurrence. Whether these are interesting depends on your application and what you want to know about the system's operation:

Asymmetric occurrences - Cxy only gets incremented if y occurs with 30 minutes after x. If y occurs before x, Cyx gets a bump, but not Cxy.

Time-dependent correlations - Instead of increasing Cxy by 1 iff y occurred within 30 minutes of x, increase Cxy by $e^{-t/\tau}$, where $\tau$ is some time constant corresponding to an expected Poisson distribution - and $t$ is the time between when x is on and y is on. Alternately, if you want to test whether there's a peak in correlation separated by about 20 minutes, you can use a chi-squared weighting for Cxy and scale your time so that 20 minutes is at the peak of the distribution.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .