I'm a psychologist studying perception of visual ensembles (e.g., lots of lines with different orientations drawn on a screen) that have different underlying probability distributions. One of the reviewer's for our paper has asked us to justify a statement that seems intuitively correct to me but I wasn't able to find a proper reference. The statement in question says that a mixture of two uniform distributions is more complex than a normal or a uniform one. The mixture distribution here consists of two non-intersecting uniform distributions with equal ranges but different means. Intuitively it seems to me that it should be more complex as its probability density function has more parameters than the functions of a single uniform or a normal distribution hence it could be said that it has lower "description length".

Am I correct in saying that this mixture distribution is more complex? And if so, could you please provide any reference supporting this?

  • $\begingroup$ Do you have a precise definition of "more complex" or is this just an informal assertion? If the latter then I'm not sure what sort of mathematical reference you would be looking for. $\endgroup$ – Rahul Dec 27 '16 at 17:05
  • $\begingroup$ The problem is that I'm not familiar with mathematical definitions of complexity for probability distributions. I was trying to get through this paper arxiv.org/pdf/1511.00529.pdf that has a definition of complexity and examples for uniform and normal distributions, but I am not able to extend it to the mixture distribution (and I'm also not sure I understand it fully). $\endgroup$ – Andrey Chetverikov Dec 27 '16 at 17:19
  • $\begingroup$ @AndreyChetverikov OK, you are not familiar with mathematical definitions of complexity for probability distributions. But, in what sense did you claim in your paper that one distribution is more complex than the other? $\endgroup$ – Zoran Loncarevic Dec 27 '16 at 17:28
  • 1
    $\begingroup$ As I explained in the post, it seems to me that it should be more complex as its probability density function has more parameters than the functions of a single uniform or a normal distribution hence it could be said that it has lower description length. And minimal description length seem to be relatively comprehensible notion to me (however problematic it is to pinpoint it). It also seem to be more complex to me in a sense that there are more inflection points in probability density function. $\endgroup$ – Andrey Chetverikov Dec 27 '16 at 17:43
  • $\begingroup$ Is all you're saying that the distribution is more complex because it has a longer natural-language description? $\endgroup$ – Michael McGovern Dec 27 '16 at 18:14

I am puzzled about your distributions. In the Question you say 'uniform', but in one of the Comments you say 'normal'. I suspect you may mean 'normal' throughout.

Supposing that you mean 'normal' distributions, here is an Answer to your question: If the means $\mu_1$ and $\mu_2$ of two normal distributions are separated by several standard deviations, then their mixture is bi-modal (has two 'humps').

For a specific example, suppose the two normal distributions are $Norm(\mu_1 = 80, \sigma = 10)$ and $Norm(\mu_2=120, \sigma = 10).$ Also suppose that the first distribution is randomly selected 60% of the time and the second 40% of the time.

Below is a simulation in R statistical software that simulates this experiment with $n = 500$ subjects making random choices between the distributions as above to get values $Z_i$ for $ i = 1, 2, \dots, 500.$

n = 500;  x = rnorm(n, 80, 10);  y = rnorm(n, 120, 10)
choice = rbinom(n, 1, .6)
z = choice*x + (1-choice)*y
hist(z, prob=T, ylim=c(0,.03), col="skyblue2", main="Simulated Mixture of 2 Normals")
  curve(.6*dnorm(x,80,10) + .4*dnorm(x,120,10), lwd=2, col="blue", add=T)

The density curve shown is found as follows: Let $\varphi_1(z)$ and $\varphi_2(z)$ be the density functions of the two normal distributions. Then the density function of the mixture distribution is $$\varphi_{\text{mix}}(z) = .6\varphi_1(z) + .4\varphi_2(z).$$ Each simulation run will produce a slightly different histogram, but the density curve is the same as long as the parameters remain the same.

enter image description here

If the means of the two normal distributions being mixed are closer together, the density function of the mixture distribution may not show two distinct modes, but this does not mean that the mixture distribution is an ordinary normal distribution. The figure below shows the simulated mixture distribution for $\mu_1 = 90,\, \mu_2 = 110,$ and $\sigma = 10.$ In particular, the mixture distribution is not symmetrical.

As you suggest in your comments it is more 'complex' in that it takes four parameters to describe: the two means, the common standard deviation, and the probability with which the first distribution is chosen.

enter image description here

  • $\begingroup$ Advice to satisfy referee quibble: (1) Remove complex because you're having trouble defining it. (2) Say: "A mixture distribution need not have the same shape as the distributions being mixed. If a dist'n is a mixture of $k$ dist'ns with density functions $f_i(x)$ and respective weights $w_i$ of occurrence, then the mixture density is $\sum_{i=1}^k w_if_i(x).$" (3) Refer to Bruce Lindsay's monograph referenced at the very end of Wikipedia article on 'mixture distributions'. $\endgroup$ – BruceET Dec 28 '16 at 19:08

in one of your comments you mentioned the paper "Measuring the complexity of continuous distributions". In this paper Complexity is defined as a function of Shannon's entropy. Shannon's entropy provide a measure of the average uncertainty of a system given a probability distribution. Thus, this Complexity determines the "balance" between emergence of new patterns, and the self-organization of the system.

Intuitively, Emergence can be understood as the uniformization of a probability distribution (for instance, the white noise which has a uniform distribution has the highest Emergence). Self-organization can be understood as the concentration of probability around a specific state(s) of the probability distribution (a delta Dirac has the highest self-organization, thus, the lowest emergence since there is no change). Then, a distribution with high complexity has one or few states which concentrate a large proportion of the probability, and many others with very low probability.

Thus, using these measures, it is not conclusive if a mixture of distributions will be more complex than only 1 distribution. For instance, compare a power-law distribution with the second mixture provided by @BruceET. Given a suitable x_min value, the power-law will surely display more complexity than the mixture of normals, since the latter is more equiprobable than the former.

Best regards, Guillermo

  • $\begingroup$ Thanks for the answer and especially for the intuitive explanation of the paper! $\endgroup$ – Andrey Chetverikov Feb 23 '17 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.