Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Note - I put piecewise in quotes because I don't think it's the right term to use (I can't figure out what to call it).

I am building a program to model the load that a user places on a server. The load a user produces follows a normal distribution. Depending on which application the user is using, however, the mean and variance of that normal distribution will be different. What I am trying to do is calculate the overall mean and variance for a user's activity given the proportions of time they use each application.

For example, Application A follows $\mathcal{N}(100, 50)$ and Application B follows $\mathcal{N}(500, 20)$. If a user uses A 50% of the time and B the other 50%, what is the mean and variance of the data that the user would produce during a day?

I'm able to simulate this by selecting a number from a uniform distribution between 0 and 1 and then generating a value from the appropriate distribution. Something like this:

$f(x) = \begin{cases} \mathcal{N}(100, 50), &0 \le x \lt 0.5\\ \mathcal{N}(500, 20), &0.5 \le x \lt 1\\ \end{cases} $

When I simulate a large number of these values and measure the results, it looks like the mean is just

$\sum\limits_{i=1}^n\mu_ip$

where $p$ is the percentage of the day a user is using each application.

I can't figure out what pattern the variance follows or what the formula might be to determine it without measuring a bunch of simulated values (When I simulate the above example, the variance looks to be something close to 41500).

I'd appreciate confirmation that how I'm calculating the combined mean is correct and some help in figuring out how to determine the variance of the overall distribution.

share|improve this question

3 Answers 3

up vote 2 down vote accepted

Let the two normal random variables be $X$ and $Y$, where $X$ is chosen with probability $p$, and $Y$ is chosen with probability $q=1-p$.

If $W$ is the resulting random variable, then $\Pr(W\le w)=p\Pr(X\le w)+q\Pr(Y\le w)$.

Differentiate. We get $f_W(w)=pf_X(w)+qf_Y(w)$.

The mean of $W$ is $\int_{-\infty}^\infty wf_W(w)$. Calculate. We get $$\int_{-\infty}^\infty w(pf_X(w)+qf_Y(w))\,dw.$$ This is $pE(X)+qE(Y)$, confirming your observation.

For the variance, we want $E(W^2)-(E(W))^2$. For $E(W^2)$, we calculate $\int_{-\infty}^{\infty} w^2(pf_X(w)+qf_Y(w))\,dw$. This is $pE(X^2)+qE(Y^2)$.

But $pE(X^2)= p(\text{Var}(X)+(E(X))^2)$ and $qE(Y^2)= q(\text{Var}(Y)+(E(Y))^2)$

Putting things together we get $$\text{Var}(W)=p\text{Var}(X)+q\text{Var}(Y)+ p(E(X))^2+q(E(Y))^2- (pE(X)+qE(Y))^2.$$

Remark: For a longer discussion, please look for Mixture Distributions.

share|improve this answer
    
So, to put it more generally: $\text{Var}(W) = \sum\limits_{i=1}^np_i(\text{Var}(X_i) + E(X_i)^2) - (\sum\limits_{i=1}^np_iE(X_i))^2$ –  Adam Platt Aug 5 at 17:57
    
Yes, the same argument works for $X_i$ with probability $p_i$. One could generalize further, to sums to $\infty$, also to "continuous" mixtures. Mixtures happen a lot in Statistics, since populations can often be stratified. –  André Nicolas Aug 5 at 18:15
    
Perfect. Thank you so much for your help! I already ran some simulations with a few different combinations and it works beautifully. –  Adam Platt Aug 5 at 18:47
    
You are welcome. Fast work. –  André Nicolas Aug 5 at 19:49

What you are wanting are the distributional quantities of the mixture distribution. There are some convenient formulas to do this: let $T$ be the unconditional load a user places on the system, and let $X$ be a Bernoulli random variable that indicates whether a user is on system $A$ or $B$. So $A = (T \mid X = 0) \sim \mathrm{Normal}(\mu_A = 100, \sigma_A^2 = 50)$, and $B = (T \mid X = 1) \sim \mathrm{Normal}(\mu_B = 500, \sigma_B^2 = 20)$. That is to say, $$\begin{align*} \mathrm{E}[T \mid X = 0] &= \mu_A, \\ \mathrm{E}[T \mid X = 1] &= \mu_B, \\ \mathrm{Var}[T \mid X = 0] &= \sigma_A^2, \\ \mathrm{Var}[T \mid X = 1] &= \sigma_B^2. \end{align*}$$

Then by the law of total expectation, $$\begin{align*} \mathrm{E}[T] &= \mathrm{E}[\mathrm{E}[T \mid X]] \\ &= \mathrm{E}[T \mid X = 0]\Pr[X = 0] + \mathrm{E}[T \mid X = 1]\Pr[X = 1] \\ &= \mu_A (1-p) + \mu_B p,\end{align*}$$ where $p$ is the probability that a user is on system $B$. The variance is calculated by $$\begin{align*} \mathrm{Var}[T] &= \mathrm{E}[\mathrm{Var}[T \mid X]] + \mathrm{Var}[\mathrm{E}[T \mid X]] \\ &= \mathrm{Var}[T \mid X = 0]\Pr[X = 0] + \mathrm{Var}[T \mid X = 1]\Pr[X = 1] + \mathrm{Var}[\mathrm{E}[T \mid X]] \\ &= \sigma_A^2 (1-p) \sigma_B^2 p + \mathrm{Var}[\mathrm{E}[T \mid X]]. \end{align*}$$ The last term requires a little subtlety to understand. The variable $\mathrm{E}[T \mid X]$ is a generalized Bernoulli, which takes on the value $\mu_A$ with probability $1-p$ and $\mu_B$ with probability $p$ (rather than $0$ and $1$). So we may write this as $$\mathrm{E}[T \mid X] = X(\mu_B - \mu_A) + \mu_A,$$ where $X \sim \mathrm{Bernoulli}(p)$. Therefore, $$\mathrm{Var}[\mathrm{E}[T \mid X]] = \mathrm{Var}[(\mu_B - \mu_A)X + \mu_A] = (\mu_B - \mu_A)^2 \mathrm{Var}[X] = (\mu_B - \mu_A)^2 p(1-p).$$ Hence $$\mathrm{Var}[T] = \sigma_A^2 (1-p) + \sigma_B^2 p + (\mu_B - \mu_A)^2 p(1-p).$$ The general case with $n$ systems $S_i \sim \mathrm{Normal}(\mu_i, \sigma_i^2)$, $i = 1, 2, \ldots, n$, where the user is on system $i$ with a probability of $p_i$, with $\sum_{i=1}^n p_i = 1$, involves a categorical distribution rather than a Bernoulli.

share|improve this answer

Recall that $V[X]=E[X^2]-E[X]^2$. Hence if you are given $E$ and $V$ of $X_1,X_2$ and combine them as described to a new random variable $Y$ by using a 0-1-random variable $Z$, i.e. picking $X_1$ if $Z=1$ and picking $X_2$ if $Z=0$, we find $$ E[Y]=P(Z=1)\cdot E[Y|Z=1]+P(Z=0)\cdot E[Y|Z=0]=pE[X_1]+(1-p)E[X_2].$$ By the same argument we find $$ E[Y^2] = pE[X_1^2]+(1-p)E[X_2^2]$$ Substituting $E[X_i^2]=V[X_i]+E[X_i]^2$, we obtain $$ \begin{align}V[Y]&=E[Y^2]-E[Y]^2 \\&= p(V[X_1]+E[X_1]^2) + (1-p)(V[X_2]+E[X_2]^2)-(pE[X_1]+(1-p)E[X_2])^2\\ &=pV[X_1]+(1-p)V[X_2]+p(1-p)(E[X_1]^2+E[X_2]^2)-2p(1-p)E[X_1]E[X_2]\\ &=pV[X_1]+(1-p)V[X_2]+p(1-p)(E[X_1]-E[X_2])^2.\end{align}$$

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.