2
$\begingroup$

I have these entries in my database that looks like this:

ItemID  Coefficient    Time 
14      1.74           "2014-12-21 17:13:29"
14      1.73           "2014-12-21 17:13:30"
78      2.55           "2014-12-21 17:17:11"
78      2.56           "2014-12-21 17:18:12"

For each item I have an entry for each second for a a period of ~5 minutes. So each item has ~300 data entries. All in all I have thousands of items, therefore lets say 300 000 entries.

I would like to know in average, is there a tendency that the Coefficient within these 5 minutes will rather drop or increase over all these items and how strong is this tendency.

I can see it visually that one items coefficient does fall or increase, but what algorithm should I use to find it out over all of these items?

I am not strong in maths, but I was googling about it, and found out about regression, but not sure how to exactly use it.

Can you give me a hint or suggestion, which algorithm I should use to analyze this data?

$\endgroup$
0
$\begingroup$

The resulting price of $i^{th}$ item is a sum (or product, or some another combination) of many random variables. The only way to define the bounds of possible result is to find probability distributions of the most meaningful ones and guess about how are they connected using some tests.

Statement: N objects, each one have almost continuous function $P_i(t)$ where $P$ - price, $t$ - time.

Question: what can we say about $P_i$ and how it changes over time?

I see 3 possible solutions:

  • Define several most important random variables $(v_1,v_2,...)$ and try to find $f_i$ such that $P_i = f_i(v_1,v_2,..)$ where $P_i$ - price of $i^{th}$ item. You can do it by making/refuting hypotheses (long iterational process). Don't forget, $v_j$ may change over time periodically (for example, peoples usually start buying useless stuff in december 20-31). This is a way of R&D (never ends and may not give concrete results).

  • Find some mathematical model in papers/blogs/conferences materials and try to apply it for your data. Choose model from related sphere, there is no some good standart approaches for all possible data, because online shop differs from shoes store. It's possible to find some approximations/facts that may help . For example - suppose we know how many visitors online store has and how many purshcases were made. What can we say about visual design efficiency? I'm sure someone has already made some research steps to find it out.

  • Find a company or analytics service, pay some $, and they will do both prevous steps for you (most probably, not so good as you expect).

=================================================

Regression is the method to estimate the dependence of random var $Y$ from $X$ using some mathematical model + some amount of data. For example linear regression $$Y = K*X + B$$ where $X$ - problem difficulty, $Y$ - time needed to solve it. Find good values for $K$ and $B$ by least square method using your data. Check you model by suitable statistical test (Fisher's or Pearson's tests) and make a conclusion whether linear model is adequate (in this case - no, there must be qudratic or even cubic dependence). Moreover, maybe $Y$ is the result of two or more random variables $Y = f(X,Z,..)$.

$\endgroup$
  • $\begingroup$ I was a little unclear in my first post. The number is not actually a price, it is more like a coefficient. If you are familiar with sportsbetting, then there are coefficients. And in sports, inside 5 minute gap, I want to find out tendencies. So this removes the factor of people buying items on christmas. Does this make the task a little easier? $\endgroup$ – Jaanus Dec 21 '14 at 19:01
  • $\begingroup$ Also, what exactly do you mean under defining random variables? I don't seem to have anything random here, just plain data and trying to analyze it. $\endgroup$ – Jaanus Dec 21 '14 at 19:03
  • $\begingroup$ @Jaanus as I understood, you want to do effective bets. There is no way to do it effectively on such small period of time (every 5 minutes) having only data about other bets. Mb you saw the film "Moneyball". There was a baseball team coach and young matematician who decided to choose new players by their statistical characteristics, not by contract price or good impression. That's exactly what you should do. $\endgroup$ – Ralor Dec 22 '14 at 1:50
  • $\begingroup$ @Jaanus Player is a set of random variables (see link above for definition) $C = \lbrace c_i \rbrace$. $c_i$ may be, for example - football (soccer) player efficiency in attack against 2 players. Every team has it's style of playing, which makes some $c_i$ more meaningful, it can be described as "weight" for every characteristic $W = \lbrace 0<w_i<1\rbrace$. Sometimes they change their style (just a little) to play better after exact team. So for effective statistical predictions about 2 teams game you need data $C$ about every player, $W$ for every team and some method to combine this data. $\endgroup$ – Ralor Dec 22 '14 at 1:51
  • $\begingroup$ @Jaanus Those who know how to do effective bets without stupid luck never discuss it, so don't ask anyone about details. In case he know - he would never say the truth. In case he don't know - you would learn something useless or pay your money for nothing. $\endgroup$ – Ralor Dec 22 '14 at 1:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.