Finding meaningful differences in a set of time measures An individual is presented with a list of words, one at a time, and asked to answer the first thing to comes to her mind. The delay of all answers is recorded. So I have a list of words, each paired with the time it took for the individual to find an answer (response delay). 
Some psychological studies indicate that words with a longer or shorter than average response delay have some special meaning in the subject's psychological profile. Despite the fact that this statement could be right or wrong, I'm interested in the best mathematical approach to find such words that exhibit a "longer than average" or "shorter than average" response time.
So far we've been just taking the average response time, and setting some fixed percentage of delay we consider long or short enough. This is of course a very simplistic approach. I think I could apply some standard hypothesis tests to answer the question: what are the odds that this response delay is equal to the average? and keep those results with a low p-value, but I'm not very versed on statistics, so I don't know the exact test I would need here.
Also, I'm interested in a slightly more complicated version, where the subject is presented with two list of words intermingled. The words from the first list are considered neutral, and are expected to have more or less the same response delay. However, the words from the second list are loaded, which means we expect they to hit something in the subject's subconscious part, and so he might have longer or shorter response delays. I would like to take the times from the first list to make a baseline for what are considered average response delays, and then check the times in the second list, looking for longer or shorter than expected response delays. What kind of test would I need here?
 A: Statistical inferences about whether a value is too high or too low require knowledge of probability distribution. The hypothesis based on pyschological studies is that different words result in different response delays. And you want to identify words with more than average or less than average delay.
As a starting hypothesis you can assume that each word is associated with a fixed response time. There is a distribution of these times for the entire population of words that you do not know but can estimate using the response times you observe in the experiment. You can do this estimation separately for each subject or can increase statistical power by estimating the distribution using all subjects' response times if the distributions of different subjects are related in some way such as shifting of means across individuals while retaining the shape of the distribution.
The most efficient estimation procedure will depend on what is already known about how response times are related across subjects. Without any assumptions, one subject's response times are completely unrelated to another subjects' response times so there is no benefit from pooling. If there is some link, pooling may help. In pooling case, the estimation may retain information about which response time corresponds to which word or just use the set of response times for each subject. The identity of words is important in estimation if (i) words used have significant overlap across individuals and (ii) the ordering of words according to response times is expected to be similar across subjects. If the answer to any of these questions is no, it is appropriate to use the response times without worrying about which words produced those response times.
The best estimation procedure may require trying many specifications to see which fits data best. Suppose $t_{ij}$ is the response time of subject $i$ for word $j$. Here are some possibilities:


*

*$t_{ij} - \mu_j$ follows a distribution $f$. You could perform a quantile regression with a dummy variable for each subject to estimate $\mu_j$. Then all $t_{ij}-\mu_j$ values pooled together represent $f$ and you can find $p$th percentile value from this distribution and adjust it for subject $j$ by adding $\mu_j$. Alternatively, instead of a regression just use $\mu_j$ = average of all response times for subject $j$ to estimate $f$.

*$(t_{ij} - \mu_j)/s_j$ follows a distribution $f$. You could estimate $\mu_j$ and $s_j$ with sample mean and standard deviation of subject $j$ and then use values $(t_{ij} - \mu_j)/s_j$ to estimate $f$.

*If identity of words is important, dummy variables can be included for all words with dummy variable $d_j$ equal to $1$ for word $j$ and $0$ otherwise.

*You could use mixed level methods to allow for shifts in mean and standard deviation that depend on word or subject. A detailed description for Stata package is at http://www.stata.com/manuals13/me.pdf. See page 285.

*The distributions of subjects may be related in more complicated ways that are not captured in such simple transformations. Suppose some transformation of response time, $g(t_{ij}) - \mu_j$ may follow distribution $f$. Then, all times must first be transformed to $g(t_{ij})$ before estimation. If $g$ is known such as $g(x)=x^2$, then this is trivial.


These above approaches seem complicated compared to the typical normal distribution assumption. The reason is that we are estimating the distribution of response times across words which is fixed, unlike the distribution of error in mean response time, which approaches normal distribution with large samples.
To estimate distribution from your experiment, you need to have a large enough sample. Suppose you assume a distribution $f$ with only one unknown parameter that is the mean $\mu$. Suppose response time for word $i$ is $y_i$. Then you are asking that if the mean response time across all words equalled $y_i$, how likely are the response times you observed? You calculate the probability using the assumed distribution with mean $\mu=y_i$. This probability may be too low if most response times are above $y_i$ or most are below $y_i$. For many distributions, the test depends on the observed mean rather than all response times observed so if $y_i$ is too much above or below sample mean, you can flag the word as having higher than average or lower than average response time. How much is "too much" depends on the distribution.
Practically, with enough observations, Central Limit Theorem ensures that sample mean is normally distributed around average. Any statistical package will let you determine the p-value that the mean of the population is more than $y_i$ or less than $y_i$. You can manually do this as follows.
Suppose with $n=30$ words, you get response times: 10,9,12.2,8.9,10.5,10.6,8.2,10.8,11.2,10,11.7,10.4,7.2,11.6,7.3,13.5,9.3,10.2,11.8,13.6,9.5,10.5,9.1,10.8,12.5,10.2,11.1,8,9,10.6
The average is $\mu=10.31$. The sample variance $s^2$ is (sum of squares of all times - $30*10.31^2$)/(30-1) = 2.57. If the p-value you are looking for is 5%, determine the value at which right-tailed probability equals 5% in F-distribution with 1 and 30-1=29 degrees of freedom (Excel's function finv). This value equals $x=4.18$. Then all $y$'s less than $\mu-\sqrt{x s^2/(n-1)}=9.70$ are less than average with p-value of 0.05 and all $y$'s with response time more than $\mu+\sqrt{x s^2/(n-1)}=10.92$ are greater than average with p-value of 0.05.
The problem with this approach is that as you use more and more words, you will estimate average more precisely and statistically you will be able to classify almost all words as above average or below average. This is probably not what you want. A better approach may be based on findings from psychology domain about how much difference from mean response time is significant. To some extent this is a question of how you define words to have a special meaning in the subject's psychological profile. To find more special words, you should use a narrow definition of special meaning and consider large variation from mean time.
If the focus is not on comparison of response times across individuals, one could classify a fixed fraction $f$ of all words with extreme response times as having special meaning for the individual's psychological profile. Of course, the thresholds won't be comparable across individuals if some people have a lot more special words than others. No fancy statistics is required in this case. You could take fraction $f/2$ words with least response times and fraction $f/2$ with greatest response times. Or you could take fraction $f$ with greatest absolute deviation from mean time.
Another issue is that so far I have assumed that each word is associated with a fixed response time and differences in response times arise only because words differ. But there may be some randomness in response times. That is, the same word may result in different response times if the experiment could be repeated many times without complications from memory or repeat experiment issues. In that case, a response time may be high either because the word has special meaning for the subject or because this happened to be one of those random times when the response time is high. Even in this situation, nothing changes in identifying words with extreme response times. However, inferences about special words will be weaker for individuals whose response times tend to be more random. Let us assume that the randomness in response times is same for all kinds of words. Suppose somehow we know that this randomness is respresented by standard deviation of  $s$, probably by providing the same word to the individual many times . Then, a word's average response time is below mean with probability 2.5% if single response time in the experiment was less than the sample average response time minus $1.96s$.
Your second experiment seems to be on the line of the method in the previous paragraph. You can estimate mean $\mu$ and $s$ from the first set of words so words in the second set with response time less than $\mu-ks$ or more than $\mu+ks$ will have special meaning where $k$ is a positive constant you can choose to match a p-value.
