I have some frequency data that I have gathered from the web (sourced from a video game) and stored in a database.

At the moment, the data consists of around 180,000 rows which will probably grow quite quickly. The data (each row) itself consists of an 'item' and its properties along with a 'user' that the item belongs to and I've been thinking about how to rank rate/value these items into value bands.

The items carry a monetary/market value but I cannot gather this data en-masse. Therefore I would like to attach my own estimated points system that approximates market value based on frequency alone, and other properties that the item has. To clarify, the higher rated an item is the more rare and valuable it should be. (The data I have so far is the rarity/frequency). I'm thinking that some analysis techniques would be required.

At the moment there are roughly 4000 distinct items with frequencies ranging from 1 (very rare) to 1500 (very common) amongst this collection of 180,000 (pulled from ~1200 users).

I'm looking for analysis techniques that might be able to bring together these properties to calculate a point value for any given item. (If this can be done)

(Also if nothing else I'll just use low frequency = high rating)

  • $\begingroup$ So why not rank your items according to frequency? Rank 1 to item with lowest frequency, rank 2 to the one with second highest frequency etc? You only have to keep one row per item to keep track of the cumulative frequencies, to be updated as new data comes in. For better answers you probably have to give us more information. In particular, why should a ranking based on frequency alone correspond to a ranking based on the value of these items? What is the ranking supposed to achieve: rank according to value or frequency or something else? $\endgroup$ – Nameless Nov 6 '14 at 0:40
  • $\begingroup$ Hi @Nameless after thinking about this you're right frequency is not enough alone to measure its market value although it does seem to hold some significance. There are other properties of each item which may possibly have some effect on the market value but it would need to be analysed to see. Will update the question with more info.. $\endgroup$ – McT Nov 6 '14 at 11:29

What you could use is called hedonic pricing. The idea is that you estimate the value of each item based on its characteristics. I don't know which kinds of item you have, but suppose you have cars. Each car has a value depending on age, color (red or white), make (Mercedes or Toyota), and whether it is a convertible. These could be any other properties these items have and you have in your dataset. What you can do now is run a linear regression of the kind $$Value=b_0+b_1*Age+b_2*Red+b_3*Mercedes+b_4*Convertible+e,$$ that is, you estimate a function (by finding parameter values b0-b4) that predicts the value of the car based on its characteristics - here age, make and so on.

Suppose after estimation it turns out that $b_4=1000$. That would mean that a Mercedes car - all else equal (same color, same age, also convertible) - is $1000 worth more than a Toyota. These kinds of predictions can be quite crude with only a few characteristics to predict with, but if you have enough it becomes quite precise. I explained this method in a bit more detail here, but linear regression is quite common and you can read up on it everywhere.

Your rating would then be computed using the equation you estimated, just plugging in the variable values of the properties (i.e., which age the current car has, which color, make etc.). It might look like this: $$ValueRating=10000-1500*Age+100*Red+4000*Mercedes+5000*Convertible,$$ where Age is in years, Red is 1 if the car is red and 0 if it is white, Mercedes is 1 if and only if it is a Mercedes and 0 otherwise, and Convertible is 1 if and only if it is one.

This method makes sense if you have enough properties that can be used to predict the item value. To estimate the equation you need a data sample where you observe the value (the 180k you have right now would probably be enough). For any new data that comes in, you don't need the item value any more, you can predict it based on its properties (which you have to observe). I don't know if this is useful in your specific case (more information would be helpful), but it allows you to reduce each "row" as you call it to a single number (ValueRating) or two, if you want to keep an id.

| cite | improve this answer | |
  • $\begingroup$ This is very insightful, looking forward to trying this out. $\endgroup$ – McT Nov 7 '14 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.