2
$\begingroup$

From a mathematical point of view, what would be the most correct way to calculate the correlation, Pearson, of two temperature measure devices that are not synchronized.

The first one is giving the temperature each hour. It's external so relatively stable.

The second device is giving an internal temperature with some chaotical measure points (sometimes 10 points in 15 minutes, after 1 hours with nothing). We can assume here the temperature might change quickly.

It's a bit more tricky than it looks like (at least to me), thanks for any help, ideas or hints

$\endgroup$
1
$\begingroup$

Well, you do not tell much about the scope you aim to, the quantity of data you have, etc. Talking in very general terms, before maths I would put physics.
1st point is: on physical grounds, do you expect the internal temperature to follow, with a time lag, the external one ?
2nd: is the lag much greater, or much shorter than the observation period (1h) ?
3rd: is the "chaos" due to what? might it have a long period component or not ?

Depending on the "evaluations" above you might proceed with a plain Pearson correlation, or with a time crossrelation analysis, or with a Fourier decomposition and filtering, etc.

addendum as per your comment

"How to calculate the correlation of two measures that are not taken at the same time, one with more points than the other, with holes."

So, given the situation that you describe, I would approach the problem as follows:

  • more points / holes
    Take the irregular sequence, and interpolate. Since temperature is supposed not to vary abruptely, a spline or trigonometric interpolation should be appropriated. Which to choose, on which sub-intervals to apply, and up to which degree, is to be deducted from the actual data available (and from the physical background..). You get a y(t).
  • comparison
    Now you can take the more regular sequence ($x(n \tau)$), and generate the list of 2-D points $\left(x(n\tau),\,y(n\tau) \right)$. On this it makes (some) sense to calculate the Pearson correlation.
    If the $x$ also has some holes, you can interpolate it as well, to get a $x(t)$.
    And once you have $x(t)$ and $y(t)$ you may perform (if you wish) a discrete or continuous time cross-relation, which is practically a correlation computation repeated for different values of time lag (for $\Delta t =0$ you get the correlation before): so it might be in any case more rich of information.
$\endgroup$
  • $\begingroup$ One it's an aircon, the other the external temperature. But windows get open ,doors get open, the aircon might be off or not working. You expect them not following each other, in theory. The meters give more information points, sometimes they don't work for hours but on average it's a few by hours (yes it's a mess). Too complicated to think about a lag, as we have a lot of meters. IMHO, it's best to think this from a mathematical point of view: How to calculate the correlation of two measures that are not taken at the same time, one with more points than the other, with holes. $\endgroup$ – ic3 Sep 3 '17 at 16:20
  • $\begingroup$ @ic3: took note of your comment and added a reply as addendum to my previous answer. $\endgroup$ – G Cab Sep 3 '17 at 17:51
1
+100
$\begingroup$

Here's just a quick shot off the hip with a few thoughts and ideas.

In any case, it is difficult to give a reliable answer without having looked at the actual data in some depth: which approaches are likely to work and be practical will depend on what the data actually looks like, in particular how fast temperatures vary in each of the temperature series.

1. Interpolation

The first, and simplest, is the suggestion already made by G Cab, to use interpolation to fill in the gaps. If one point has more frequent measurements, that might be the one you'd do the interpolation on.

However, if the measurements are sufficiently far between that the temperature may have varied substantially during the period, this might not be a reliable approach. The problem is that the interpolated values will typically tend to be more stable than the real measurements: at least if they are some kind of weighted mean of the closest measurements.

2. Interpolation using time-series models

You can try to set up a time-series model to model how the temperatures vary. You can then assume you have measurements made at given points (which may well be random/irregular), and try to estimate the actual temperature.

On kind of method for doing this kind of estimation is the Kalman filter, although I guess you can find similar solutions (which may have different names).

In many respects, this is similar to the interpolation, except you have a more advanced method for doing the interpolation. You can also take into account that accuracy of the estimates so that the points in time when you have accurate estimates are given greater emphasis in the computation of the correlation.

3. Correlated time-series models

You can go one step further with the time-series models and assume that there are two correlated time-series with measurements taken at arbitrary time points, fit the time-series, and try to estimate the correlation based on the model.

This is similar to point 2 except instead of actually interpolating the temperatures and computing correlations on the interpolated values, you fit a more complex model and estimate the correlation between the two models.

In any case

In any case, you should try to make different models for temperature series and for the timing of measurements in order to generate random data for which you know the actual correlation, and then test your methods on those. By "model", I here mean just some method of generating the temperature series (for all time points so you can compute the actual correlation!) and then sampling the random measurement points to see how well you can estimate the actual correlation.

$\endgroup$
  • $\begingroup$ I do not see much a difference between interpolating and "creating a temperature model": none can give you the missing measurements, if sudden random changes are expected to occur ! That the knowledge of the physical conditions (your "model") are essential to give any meaning to the correlation is what I premised to my answer, but OP replied that internal measurements are subject to quite random and unrepeated phenomena. So I do not see in your answer any added value. $\endgroup$ – G Cab Sep 4 '17 at 13:42
  • $\begingroup$ @GCab: Interpolation has a few potential problems. It will tend to be a weighted average of the closest measurements, and thus have different variablilty and correlations than true measurements taken at the desired times. By using an apporopriate statistical model, you can correct for this, and also take into account the accuracy of the estimated temperatures between actual measurements: inaccurate estimates, eg when there is a long time span between measurements, should be given less influence. Pure interpolation, in comparison, is just filling the gaps in the data and hoping for the best. $\endgroup$ – Einar Rødland Sep 4 '17 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.