Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I don't have the proper vocabulary, so I'll just try to present two situations that I think capture my question: if you were to chart "vertical leaping ability among high school students," I would think you'd get a distribution where there'd be a lot of variance in the below-median ability but less variance at the high-end because advantages between elite and quite-fit athletes are only a matter of a few inches. Another example might be "speeds of everyone riding a bike right now" where the above-median side is limited by wind resistance.

Is this sort of situation described by its own probability distribution function or does one have to preprocess the data so that the varying / resistance aspect is removed? (That is, transform "vertical leap" or "bike speed" into calories or somesuch; but what if the underlying driver(s) are unknown?)

share|improve this question

1 Answer 1

There are at least two properties of the situation that need to be captured by a reasonable probability model. One, you have positive skew. In other words, there are lots of people at the lower end of the spectrum but very few at the higher end of the spectrum. (there are lots of athletes with moderate ability but very few with high ability). Two, the variance at the higher end of the spectrum is smaller. In other words, the variance in ability among the athletes with high ability is lower compared to the variance in ability among the athletes with moderate ability.

One could in general capture the above phenomena by a positive skew distribution (e.g., the pareto). But, such a distribution may not necessarily yield a lower variance at the higher end of the spectrum. A natural way to model such a phenomena is to assume that you have two types of populations: 'novices' and 'experts'. Then you can model the distribution of novices and experts separately with their own means and variances as follows. For the sake of illustration, I will assume normality but that is not needed.

Let:

$x$ be the variable of interest (e.g.. how high a person jumps)

We can assume that:

$x \sim N(\mu_n,\sigma^2_n)$ if the person is a novice,

$x \sim N(\mu_e,\sigma^2_e)$ if the person is an expert,

$\pi$ be the proportion of novices in the population

By definition:

$1-\pi$ will be the proportion of experts in the population

Apriori you do not know if any particular athelete is a novice or an expert. Thus, the observed variable $x$ (e.g., how high a person jumps) can be modeled as a mixture of the above two distributions:

$f(x|-) = \pi f_n(x|\mu_n,\sigma^2_n) + (1-\pi) f_e(x|\mu_e,\sigma^2_e)$

Since, a linear combination of normal distributions is a normal distribution, we have:

$x \sim N(\pi \mu_n + (1-\pi) \mu_e , \pi^2 \sigma^2_n +(1-\pi)^2 \sigma^2_e)$

You could then use maximum likelihood or the EM algorithm to estimate the parameters of interest. Suppose that your population is such that:

$\mu_e > \mu_n$ (i.e., experts are better than novices),

$\pi > 1-\pi$ (i.e., you have more novices thane experts) and

$\sigma_e < \sigma_n$ (i.e., experts have lower variance than novices)

In the above situation, given sufficient data your estimates will accurately reflect the above pattern.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.