Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I am using this weighted sum from the question How to build a function that gives recent years higher weight?

To reduce the value I tried square root of $X,$ then tried the $8$th root, $\sqrt{\sqrt{\sqrt{X}}}$ which is $X^{\frac{1}{8}}$ and get better results.

My question is how can I justify this use? Can I know why I got better results when using the $8$th root or is it just random, depend on data only? And how can I know if there is a better way of doing it?

share|improve this question
    
I doubt you can completely justify an eighth root for this application. Also, if all of your radicands are positive numbers, you can't really justify an integer root, either. Sometimes when modeling you just have to use constants that work empirically. You can justify that if you must use a root, your root must be greater than 1; is that what you wanted to know? –  Eric Stucky Aug 5 '12 at 21:41
    
@EricStucky Why using 8th root produce better rates and ranges than square root for example ? So "use constants that work empirically" means I should experiment until I get best results? Finally, is there a wrong model ? for example if someone test with 20th root and get some results. Can anyone claim that it makes no sense ? –  tnaser Aug 5 '12 at 23:10
    
In order to say that you get "better results", you need to have a definition of "better". What are you trying to optimize? (This is also related to the experimentation question: there may be a theoretical solution for certain optimizations) As for wrong models, not really. There are models that don't behave according to your assumptions; in this case, you wouldn't want to use integer powers of $X$ because you believe that recent events should be weighted higher and this would not do that. If you assume a model should do something, and then you discover your model does not do that... –  Eric Stucky Aug 6 '12 at 0:22
    
…then you could claim it made no sense if the assumptions have a meaningful basis in reality. And if you are trying to optimize something, there are of course sub-optimal models. –  Eric Stucky Aug 6 '12 at 0:23

1 Answer 1

up vote 1 down vote accepted

There is extensive literature on data transformations, specifically to lower the variance of the residuals contained in the data, fit to a given model.

I'm not certain if it is exactly what you need, but the square root transformation is a well-known variance stabilizer.

From a cursory review of your data, I think you might be able to look into various types of regression analysis, and that would lead you into all sorts of ways to model your data, all of which have rich and rigorous justifications for such transformations.

Even more simply, you could use an exponentially-decaying rate to arrive at your weights.

The point to remember is that with your current methodology, you are just shrinking the values by multiplying them by smaller and smaller numbers. If you wanted a really small number you could use $x^{1/100}$ but you are correctly deducing that there isn't much of a justifiable reason (mathematically) to do so.

Keep experimenting with things...for one, I would try weighting each score by a factor of $x^{\frac1t}$ where t is the number of years ago. For example

Year 1: 10 movies, 5 comedies Year 2: 8 movies, 2 comedies Year 3: 10 movies, 7 comedies

take your sample proportions (Year 1= .5, Year 2=.25 and Year 3=.7) and apply the following weights:

Year 1= $(.5)^{\frac13}$

Year 2= $(.25)^{\frac12}$

Year 3= $(.7)^{\frac11}$

This would down-weight past years in a more "tidy" way.

share|improve this answer
    
I like the talk about variance stabilizer,regression analysis and exponentially-decaying. Any suggested books or tutorials will be appreciated.The problem with numbers between 0-1 , is that when you take their sqrt or (X)1^t , their value will increase. The .5 will be bigger than .7 after transformation. –  tnaser Aug 7 '12 at 11:42
    
Most likely your best bet is just to Google "variance stabilizing transformations". A good start would be here: stat.ncsu.edu/people/bloomfield/courses/st762/md-02-2.pdf ... Also, you can often times use trig functions in transformations, like sin/cos, etc. These are obviously bounded. –  Justin Aug 7 '12 at 16:14

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.