1
$\begingroup$

I hope it is not a trivial question.

I have data for a machine learning experiment from 2012 to 2018. In 2012, the samples are far different from the ones of 2018. Each sample is a little distribution and I think that the descriptive stats (like the mean, std, kurtosis and skewness) of each sample can be interesting to predict the target.

My concern is that I need to "normalize" the data, because, otherwise, the descriptive stats of a sample of 2012 would be very different of the ones of 2018, resulting in confusing data for the ML model. The issue is to be able to compare the stats between samples, not at the end of them.

¿Is somewhat impossible what I'm asking for?

$\endgroup$
0
$\begingroup$

I resolved this issue in the following way, for any one concerned:

  • As you have the raw dataset, you need to scale the data points with some metric that you could use in any of them.

For example, if you have: data = [3,6,28,19,183,99,132] --> you could use the sum (470) to scale each data point, resulting in:

data = [0.00638298, 0.01276596, 0.05957447, 0.04042553, 0.04042553, 0.04042553, 0.04042553]

As you are using the same metric (but, with different value) to scale (i.e. take proportions) every little dataset, you can then compare each of them, even if the values of years back are different.

Hope it helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.