0
$\begingroup$

I have two arrangements(i.e. permutations) of numbers. First one is the target/real arrangement. Second, is the observed arrangement.

e.g.

Target := 1,2,3,4,5,6,7

Observed := 4,1,7,3,2,5,6

Any two elements in an arrangement is not equal. What kind of test should I use?

p.s. I am not good in statistics. I am trying to evaluate a simulation model with real world data. Target arrangement is a sequence of real world events while Observed arrangement is the sequence of events which occurred in a simulation. My hypothesis is that these two are similar.

--EDIT-- I also posted this in here

--EDIT--

I have 30 samples where each sample is from a group of four people (one sequence is taken form a person). So, I have 120 sequences from real world experiment and 120 sequences from simulation results (observations). I am sorry that I tried to explain in in a simple way which made you think in a wrong way.

$\endgroup$
  • $\begingroup$ As stated, you cannot do a hypothesis test on this data. First, your sample size is 1, so its woefully underpowered. Secondly, you have not defined the set of likely target values associated with your observed values. If you are using your simulation to model a real phenomenon, then you need to know the expected discepencies between Target and Observed assuming your model is accurate. Without this, how can you evaluate if your simulation values are close enough to the Target values? Third, what defines "success" for your simulation? How are you measuring accuracy? $\endgroup$ – user76844 Dec 12 '13 at 5:45
  • $\begingroup$ Actually i have 30 samples (30 subjects participated). All target values and observed values for a particular sample are in the same range where range is [1,n] and n ~= 15. $\endgroup$ – Deamonpog Dec 12 '13 at 7:26
  • $\begingroup$ OK, that's enough to do something with then. What should the observed numbers look like if your model were correct? For example, if a target string were 2,4,6,8,10,12,14...what do you expect your "observed/simulated" string to look like? Would it ideally be an exact match? $\endgroup$ – user76844 Dec 12 '13 at 18:39
  • $\begingroup$ Yes, exact match means that simulation was 100% successful. But if i get something like 4,2,8,6,12,10,14,.. it is better than having the string reversed. Each number represents a certain event and the sequence represents a sequence of events. Two events flipping is acceptable than the event happening after a long period(larger displacements in the sequence). $\endgroup$ – Deamonpog Dec 12 '13 at 19:52
  • $\begingroup$ Ok, thanks. How are you measuring accuaracy of the simulation? $\endgroup$ – user76844 Dec 13 '13 at 17:01
0
+50
$\begingroup$

You indicated that you consider a complete reversal of the numbers worse than a couple mislocations.

Since your numbers represent events and not actual quantities, we cannot assign a simple distance measure to each pair of results (simulated vs actual) to quantify the accuracy. However, since you indicated that small mislocations in the sequenc are better than large mislocations, I wanted to suggest the following measure of accuracy:

For each element of the target sequence, find its matching value in the observed/simulated sequence and calculate the absolute value of the difference in their positions. If your simulated sequnce is missing a particular event (not sure if this is possible) then assign a value of N, where N=size of the target sequence.

Using your example:

Target := 1,2,3,4,5,6,7; Observed := 4,1,7,3,2,5,6

We would get : $Error = |1-2|+|2-5|+|3-4|+|4-1|+|5-6|+|6-7|+|7-3| = 1+3+1+3+1+1+4=14$

Now, I am not sure what you mean by a hypothesis test in this situation. You have a model and you have actual data that presumably has no measurement error, so your simulation is either right or wrong. I think what you mean is model evaluation, in which case the metric I gave above can be used to compare your current model's performance to a revised model performance.

When/if you revise your model, you can get another set of subject responses and then perform a hypothesis test using the observed Errors to determine if Model 1 is more/less accurate than model 2. That is where I see a hypothesis test being useful to you.

Let me know if the above error metric property captures how you feel about different types of errors. As it stands, the more the simulated value's location in the sequence differs from the location of the value in the Target sequence, the more error is assigned.

$\endgroup$
  • $\begingroup$ Thanks. Yes i am already using it this way. Furthermore, i found that if the ranges are the same and could be written as {1,2,3,..,k} and no repetitions occur in both sequences, then just taking the (absolute) difference of two sequences and taking the sum will give the same result as above. Futher, I am using Levenshtein difference as suggested in stat.exchange. If you get a newer idea please let me know. Since there is no similar model, i was thinking of generating random sequences and compare my results with them. $\endgroup$ – Deamonpog Dec 14 '13 at 20:44
  • $\begingroup$ @deamonpog it would have been nice to know you were using that as I asked how you were measuring similarity. Again if all you have is a prediction model and the correct answer then there is no hypothesis to test its either right or wrong $\endgroup$ – user76844 Dec 14 '13 at 21:47
  • $\begingroup$ Also I was not suggesting taking absolute differences of corresponding positions but of the positions corresponding to the same value in each sequence. That will not always yield the same value. $\endgroup$ – user76844 Dec 14 '13 at 22:16
  • $\begingroup$ @Deamonpog Here's an example: 1,2,3,4,5,6,7 vs 6,7,4,5,3,2,1 gives you 5+5+1+1+2+4+6 = 24, which is 10 more than what I calculated for your example strings. $\endgroup$ – user76844 Dec 16 '13 at 13:36
  • $\begingroup$ Yes I understood it. But what I said was that it gives the same result as taking difference.E.g., for your example it will be |1-6|+|2-7|+|3-4|+|4-5|+|5-3|+|6-2|+|7-1|= 5+5+1+1+2+4+6 = 24, which gives the same output of your suggestion. $\endgroup$ – Deamonpog Dec 16 '13 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.