2
$\begingroup$

I have a somewhat complicated question and as someone with no data analysis background, I'm searching for advice/recommendations for the best data analysis "tool" or method to do what I need.

For context, I am analyzing a phenomenon in the aurora that brightens exponentially with time. I am manually selecting events and eyeballing initial and final times to extract my brightness values ($y$-axis here). I am using ground-based imagers and extracting brightness values from an average number of pixels. I have a number of events, currently 13 which I am plotting below.

My goal is to find the growth rate of the brightness which is important for picking out the exact plasma instability responsible for this specific type of aurora. To do this, I have to only calculate the growth rate based on the linear phase of the plasma instability. In other words, when plotting log-linear, you would only see a straight line in the linear phase and when that line starts deviating, that would signify the nonlinear phase and not the same auroral phenomenon anymore.

What I have set up now is a code that puts all the curves for all the events, plots in log-linear, and calculates a line of best fit for every point after two points then after the line of best fit goes out of tolerance (i.e., below $r^2=0.85$), that next point is labeled as non-linear.

This sounds great, except that in some events, the brightness goes down for the first few time steps because I am selecting pixels in the aurora image that are getting slightly darker before the event takes place (i.e., I was sloppy at choosing my $t_0$ for the event).

Is there some kind of boxcar averaging method I can use to make sure this doesn't happen or should I apply a Gaussian filter to the data (although there aren't many data points in the time series) and then perform analysis on those?

I am also going to background-subtract all the events so that they are normalized to $0.1$ as the background value (taking the smallest brightness in the time series as the background) to make the plots a bit nicer to look at.

All events plotted linear-linear. Brightness vs. time. The linear phase is in blue, dotted green line shows the transition, and then red lines are nonlinear growth.

1

One event magnified showing more in-depth how my current algorithm is working.

2

$\endgroup$
3
  • $\begingroup$ +1, an interesting and practical question. In a somewhat different context, there might be useful ideas here. $\endgroup$
    – A rural reader
    Commented Sep 5 at 0:56
  • $\begingroup$ I believe that stats.stackexchange.com may be better suited for your question. Nevertheless, some comments: An exponential function is, by definition, a non-linear function. That is, I suggest to slightly modify your title: judging from what you wrote, you are looking for a method that tells you the point at which a line no longer approximates an exponential curve sufficiently well. As criteria you chose $R^2 < 0.85$. Another method I was thinking about would be to use local or kernel regression. These methods try to fit a smooth curve to the data. Now you could define a neighborhood ... $\endgroup$
    – Quertiopler
    Commented Sep 5 at 12:17
  • $\begingroup$ ... of size, say, $\epsilon>0$. Now for each time point $t$ you could fit a line and estimate the distance ($L_2$, $L_\infty$, ...) to your smooth fit. If it exceeds a certain threshold, you would classify the neighborhood as non-linear. You could even refine this method and only classify it as non-linear if the subsequent neighborhoods are also non-linear.$\\$Another idea would be to fit a parametrized exponential curve and estimate the parameters from the data. This method has the advantage that you are not reliant on (arbitarily) chosen classification thresholds. $\endgroup$
    – Quertiopler
    Commented Sep 5 at 12:22

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .