# How to derive thresholds from a pooled sample of values

Question:

The context of this question is actually finance, however the question itself is a statistical issue.

Suppose I have the following expression:

$$\rho = \frac{2\bar{x}}{(s^*_x)^2}+1 \ \ \ \ \ ... (1)$$

where

$\bar{x} = \frac{1}{T} \sum_{t=1}^{T} x_t$

$(s^*_x)^2 = \frac{1}{T} \sum_{t=1}^T (x_t - \bar{x})^2$

Assume $T$ is some fixed constant. Note: $T$ is just the total number of observations of $x_t$.

I have data on $x_t$ for each 'entity' (here an entity just simply refers to a firm/company). In total, I have 2228 entities and for each entity I have $T$ observations of $x_t$. Note there are no distributional assumptions on $x_t$, so one can make any reasonable assumptions in order to solve the problem that I mention further below.

For each entity, I substitute the $T$ observations of $x_t$ into Eqn. $(1)$ and obtain a value for $\rho$. Thus in total, I have 2228 values of $\rho$.

Now, a large value of $\rho$ means the entity is "bad" and a small value of $\rho$ means the entity is "good". However, the problem is how large does a value of $\rho$ have to be in order to classify an entity as "bad"? That is, what is the threshold such that if $\rho$ exceeds the threshold value, then we can classify the entity as "bad"?

For example, let's say the threshold is $400$, if entity A had a $\rho = 300$ while entity B had a $\rho = 1000$, then entity A is "good" while entity B is "bad".

My attempts so far:

My first attempt was to try get data on an entity that is known to be "bad" and then calculate its $\rho$ and use this value as the threshold. The problem is that I cannot obtain data on "bad" entities.

For my next attempt, I obtained the empirical distribution by applying a kernel density estimator on the 2228 values of $\rho$. Then I calculate the 99th percentile (for robustness, I also calculated the 97.5th and 95th percentile) of this pooled distribution and use this value as the threshold. However, the main critique is that this is too arbitrary and there is not enough rationale for using this method.

Queries:

So I am wondering if anyone has any ideas on how what statistical/mathematical techniques/methods I can apply to derive appropriate thresholds for $\rho$. Currently, I really have no idea on what tools are available for this problem.

EDIT 1

This edit is in response to some of the comments below.

I am ready (and quite confidently) to assume $\rho$ measures the "goodness" of a company. Actually I am dealing within the hedge funds industry and it is a very opaque industry. It can be shown through an arbitrage argument that $\rho$ outperforms many other standard measures of performance such as the Sharpe ratio, Sortino ratio, Jensen's alpha etc. In this sense, $\rho$ doesn't necessarily play a causal role, i.e., it is not actually concerned with deciding which hedge funds are "bad" or "good", it is in fact a measure of performance that is shown to be highly resistant to hedge fund's manipulation of their returns. It can be shown that if $\rho$ is high, then it most likely means the fund is manipulating their returns, hence the label "bad", whereas if $\rho$ is low, then it most likely means the fund is not manipulating their returns, hence the label "good".

As a consequence, I do not actually know which hedge funds are actually "good" or "bad" in my sample. I simply know the $\rho$ values of each fund, hence the need for some cut-off point that determines whether the $\rho$ values are too big 'relative' to the rest of the sample.

I am not sure if the following information will help, but it can be shown (through the use of Central Limit Theorem/Slutsky's Theorem) that if we assume $x_t$ to be a weakly stationary process with mean $\mu_x$ and variance $\sigma_x^2$, then the asymptotic distribution of $\rho$ for a single fund is given by: \begin{align*} \rho \stackrel{asymp}{\sim} N\left(1+ \frac {2\mu_x}{\sigma^2_x}, \frac 4{\sigma^2_xT}\right) \end{align*}

However, this is only the $\rho$ value for one particular fund, I am not sure how it is of any use to determining some sort of threshold for the entire sample.

Is there some sort of statistical framework/technique which we can apply here to deduce some sort of threshold? Additional assumptions are not a problem, I am just curious if we can get some sort of threshold first.

Thank you all for the inputs so far, I highly appreciate them.

EDIT 2: In response to Eupraxis1981

I ran MLE on the original log-likelihood (before the edit) which was

$$\max\limits_{\mu,\sigma^2}\mathcal{L}(\hat{\dot\rho},s^{*2};\mu,\sigma^2) = -\frac{N\ln(2\pi)}{2}-\sum\limits_{i=1}^N \{\frac{1}{2}\ln(\sigma^2+\frac{4}{s^{*2}_iT})+\frac{1}{2(\sigma^2+\frac{4}{s^{*2}_iT})} (\hat{\dot\rho}_i-\mu)^2\}$$

and got some very sensible results for the estimates of $\mu$ and $\sigma^2$. However, I am not quite sure how you arrived at the modified log likelihood currently shown, could you outline how you derived it? More specifically, how did $\dot\rho_i$ get introduced into the log-likehood? Also, what values do I use for $\dot\rho_i$? For $\hat{\dot\rho} = \{\hat{\dot\rho}_1,\hat{\dot\rho}_2....\hat{\dot\rho}_N\}$, I use the values derived from equation (1), that is, for each investment, I calculate $\displaystyle \frac{2\bar{x}}{(s^*_x)^2}+1$ using the returns of that particular investment. But what would I use for $\dot\rho_i$?

• I am not sure that I understand your definition of $\rho$. What do the multiple dots actually stand for? – Martin Van der Linden Jan 8 '14 at 16:08
• My bad, the multiple dots were to label the equation haha. Otherwise just take the definition as given (it is derived from some economics/finance reasoning which isn't important here). I should point out that there is no distributional assumption on $x_t$, so one can make some reasonable assumptions (stationarity etc). – SwiftMo Jan 8 '14 at 17:01
• You need to determine what risks your estimator is supposedly measuring, then you need to know what is an unacceptable risk, then you need to back out the value of $\rho$ that signifies that value. Once you have that figured out, you can use your sample data to make interences about $\rho$ for each entity. – user76844 Jan 8 '14 at 17:57
• Also, if you don't have any data for "bad" companies, how do you know your esimator is really measuring anything useful? If its a theoretical justification, fine, but then you should be able to derive a threshold for $\rho$. – user76844 Jan 8 '14 at 17:58
• @Eupraxis1981 I've replied to your comment with further information by editing the original post. Please see above. – SwiftMo Jan 9 '14 at 5:58

Thanks for the papers, very helpful. It appears you are trying to find a threshold for what you term the "doubt ratio" in the draft word article (i.e., $\rho: \Theta(\rho)=0)$. However, given computational difficulties, you are using an approximate form of the Doubt Ratio (i.e., $\dot\rho$) based on the second order taylor approximation. You've determined that the estimator for the approximate doubt ratio, (i.e., $\hat{\dot\rho}$) has an asymptotically normal sampling distribution as you've given in yourpost. (I've taken some liberties with the notation to keep the following explanation a littler clearer). In that case, we can re-cast the asymptotic result as:

$\hat{\dot\rho_x} \stackrel{asymp}{\sim} N\left(1+ \frac {2\mu_x}{\sigma^2_x}, \frac 4{\sigma^2_xT}\right) = N\left(\dot\rho_x, \frac 4{\sigma^2_xT}\right)$, implying that $\hat{\dot\rho}$ is an asymptotically unbiased and consisent estimator of the approximate doubt ratio, $\dot\rho$, which we are treating as a proxy to the true doubt ratio, $\rho$.

Suggestions on Approach

Given that you don't have data where you know that returns have or have not been manipulated, I think the best you can do is get a relative measure of how unusual a particular investiments risk-adjusted returns are when compared to the overall market of such investments. Therefore, the choice of benchmark population is criticial.

My suggested approach has two phases:

1. Approximate the distribution of $\dot\rho$ in your benchmark set so you can set a critical value for the true approximate doubt ratio $\dot\rho_c$,

2. Perform a one-sided Z-test on each investment to see if it's approximate doubt ratio exceeds that value.

Step 1

For this part, I would propose using a Normal Random Effects Model to estimate the variability of the approximate doubt ratios:

Let $\dot\rho\sim N(\mu,\sigma^2), \epsilon_i\sim N(0,\frac{4}{\sigma^2_iT})$ Then we can model each observation as:

$\hat{\dot\rho}_i=\dot\rho + \epsilon_i$

However, since we don't know $\sigma^2_i$, lets use its estimator, $s^{*2}_i$ to approximate the distribution of $\epsilon_i$ (i.e, $\hat\epsilon_i$) to get our final model:

$\hat{\dot\rho}_i=\dot\rho + \hat\epsilon_i$

Therefore, for a given $\mu, \sigma^2$, $\hat{\dot\rho}_i\sim N(\mu,\sigma^2+\frac{4}{s^{*2}_iT})$.

We can use maximum likelihood estimation to optimize the gaussian log-likelihood function of the observed vectors of doubt ratio estimates, $\hat{\dot\rho} = \{\hat{\dot\rho}_1,\hat{\dot\rho}_2....\hat{\dot\rho}_N\}$ and sample variances of excess returns, $s^{*2} = \{s^{*2}_1,s^{*2}_2,....s^{*2}_N\}$. This becomes the following optimization problem:

$\max\limits_{\mu,\sigma^2, \dot\rho}\mathcal{L}(\hat{\dot\rho},s^{*2};\mu,\sigma^2,\dot\rho)=-N\ln(2\pi)-\frac{N}{2}\ln(\sigma^2)-\frac{1}{2\sigma^2}\sum\limits_{i=1}^N (\dot\rho_i-\mu)^2-\sum\limits_{i=1}^{N} \frac{1}{2}\ln(\frac{4}{s_i^{*2}T}) - \sum\limits_{i=1}^{N} \frac{(\hat{\dot\rho}_i-\dot\rho_i)^2}{\frac{8}{s_i^{*2}T}}{}$ Where $\dot\rho$ is a vector of estimated "effect sizes" for each firm. This isn't strictly necessary if you just want the effects distribution, but allows you to estimate "shrinkage" estimates of each firm's doubt ratio given the observed $\hat{\dot\rho}$. You would need to optimize over an additional N variables (1 per firm) but you would get an uncertainty-weighted estimate of each firms true doubt ratio using the estimated effects distribution as a "prior" (essentially). Its slightly more Bayesian, but I thought you would want to compare the random effects estimate vs your raw estimate using the second-order approximation only.

As you can see, you will be estimating the individual doubt ratios simultaneously with the overall distribution of doubt ratios. Therefore, your initial estimation formula will give a different answer, depending on how variable the funds returns are. Your estimated distribution of approximate doubt ratios is:

$\dot\rho\sim N(\mu_{mle},\sigma^2_{mle})$, where $\mu_{mle},\sigma^2_{mle}$ are the solution to the above optimization problem. Now you can choose a cutoff. For example, $\dot\rho_c = \mu_{mle}+2\sigma_{mle}$. The rationale for this cutoff is as follows: Assume that NO manipulation is occurring in your benchmark set, in that case, you will be erroneously rejecting approximately 3% of your possible investments.

Now, assume that there are some investments whose returns are being manipulated, in that case we are assuming their doubt ratios will be higher than those that are not, on average, implying that the upper percentiles of your estimated distribution will contain some manipulated investments. Hence, the number of good investments that you reject using this cutoff will be less than 3%. Therefore, you can justify a cutoff in terms of willingness to forgo a possibly good investment (i.e., the opportunity cost is the lost returns from an exceptional investment or manager).

Step 2

I assume you know how to do a basic one sided Z-test for each investiment, $i$: $H_0: \dot\rho_i = \dot\rho_c;\space H_a:\dot\rho_i > \dot\rho_c$ for some Type I error rate.

This won't completely eliminate the arbitrary nature of the choice of cutoff, as we don't know the distribution of doubt ratios when manipulation is absent. This will merely identify the investments that have relatively high doubt ratios. For example, if 90% of hedge funds are manipulating returns, then you will only be screening out a few and labeling a lot of them as "good". Such is the case when you neither have calibration data to help determine the "manipulation free" distribution of doubt ratios nor a theoretical justification for a value from financial/economic theory.

Perhaps you can calculate the distribution of doubt ratios in a market where manipulation is unlikely using investment types known to be relatively manipulation free. Since $\Theta$ does not depend on any underlying returns distribution, it should be possible to use it to compare "apples" to "oranges" (e.g., money market fund vs hedge fund). This assumes the same type of investors are looking at each type of fund (which may not be the case, as hedge funds typically attract less risk averse people).

Anyway, perhaps something along those lines will help select a critical value. For now, lets just classify an unusual value using a percentile cutoff as above. Of course, you can always use some other firm's estimates. I think Goetzmann et al (2007) quotes Morningstar using a value of 3 in their version of the MPPM measure..you can adjust based on the observed leveraging in hedge funds (assuming you can get this info)

• Thank you for your indepth answer! That sounds like an excellent idea. I will first implement steps 1 and 2 and see how it goes, but your idea with comparing "apples" vs "oranges" is very interesting - never thought about that. I will look into it further into the literature and see whether there are investment types that are relatively manipulation free. Thanks again! I will keep in touch with you (either here or email)! – SwiftMo Jan 11 '14 at 8:47
• @user119232 glad it helped. I realized that I left my justification for the cutoff unfinished. I've filled out that section since its most relevant to your original question. Basically, the cutoff represents an upper bound on the probability that you will reject good investments just because they have exceptional returns. Its a subjective cutoff in the sense that each investor will have a different willingless to let that happen. The coverse (which is more conservative) of avoiding bad investments, can only be determined if you know the actual distribution of manipulated investments. – user76844 Jan 11 '14 at 12:08
• @user119232 hi, i realized that my log likelihood was mis indexed. I've corrected it to put the summation in the proper place. – user76844 Jan 11 '14 at 13:14
• @user119232 modified log likelihood to account for estimating random effects. – user76844 Jan 13 '14 at 20:06
• Thanks! Please refer to original post (EDIT 2) for my response. – SwiftMo Jan 15 '14 at 14:22

You said that you cannot obtain data on entities which are known for sure to be bad. So you cannot obtain an upper bound on the threshold value of $\rho$, i.e. a value $\bar{\rho}$ such that the real threshold $\rho \leq \bar{\rho}$.

But maybe you can obtain data on entities which are know for sure to be good so as to obtain at least a lower-bound on the threshold value, i.e. a value $\underline{\rho}$ such that the real threshold $\rho \geq \underline{\rho}$ ?

If you have several observations on companies known to be "good", say $(\rho_1,\dots,\rho_n)$, then you should chose $\underline{\rho} = \max (\rho_1,\dots,\rho_n)$.

That might be the best approximation you can get without further specifications of your problem or additional data on bad firms.

• This is a great idea, assuming the OP can at least identify good companies. It will be conservative, but that is never such a bad thing with investing ;-) However, it assumes that bad companies do in fact have higher $\rho$ values, which cannot be verified without actual cases of bad companies. It may be that $\rho$ is a bad discriminator. – user76844 Jan 8 '14 at 19:55
• Agreed on the last point. Reading the OP, it seemed he or she was ready to assume that the indicator accurately represent the "goodness" of the companies, i.e. that the better the company, the lower the value of $\rho$. As I understood it, this problem being assumed away, he or she was chiefly concerned with find a sensible threshold. Hopefully I wrote him or her right... – Martin Van der Linden Jan 8 '14 at 20:21
• Given the financial context calling $\rho^* = \underline{\rho}$ conservative makes sense. There is a sort of Type I/Type II error issue here. $\rho^* = \underline{\rho}$ is conservative if conservative means that we seek to avoid labeling bad companies as good ones (assuming $\rho$ is monotonic in goodness). In other contexts, $\rho^* = \underline{\rho}$ might be unconservative if a conservative strategy consists in making sure that no good company is labelled as a bad one (assuming we may not have observed the best among the good companies). – Martin Van der Linden Jan 8 '14 at 20:36
• @MartinVanderLinden Thanks. This is a good idea, however sadly I do not really know which funds are "good" either. I've replied to your comment (and answer) in more detail by editing the original post. Please see above. – SwiftMo Jan 9 '14 at 5:59
• Maybe you should migrate your question to stats.stackexchange.com to get better answers. – Martin Van der Linden Jan 9 '14 at 8:22