Let's suposse I have a vector of elements $x(n) = \{x(1), x(2), \cdots ,x(N-1)\}$ from a random process X of mean $\mu_x$ and variance $\sigma_x^2$. I want to see if I can stimate the mean and variance of the random process with my finite vector.

So, we say an estimator is unbiased if the mean of the estimator is the same as the mean of the random process. For example, the sample average is:

$$\hat{\mu_x} = \dfrac{1}{N} \sum_{n = 0}^{N-1} x(n)$$

Taking average in each member:

$$E[\hat{\mu_x}] = \dfrac{1}{N} \sum_{n = 0}^{N-1} E[x(n)] = \dfrac{1}{N} \sum_{n = 0}^{N-1} \mu_x = \mu_x$$

Thus, the sample mean is unbiased. My problem comes with the mean of the sample variance. The definition of the sample variance is the following:

$$\widehat{\sigma_x^2} = \dfrac{1}{N} \sum_{n = 0}^{N-1} (x(n) - \hat{\mu_x})^2 $$

I have to prove that:

$$E\left[\widehat{\sigma_x^2}\right] = \sigma_x^2 \left(1 - \dfrac{1}{N} \right) $$

I have tried to expand the square parenthesis:

$$\widehat{\sigma_x^2} = \dfrac{1}{N} \sum_{n = 0}^{N-1} E[x^2(n)] + E[\hat{\mu_x}^2] - 2 E[x(n) \hat{\mu_x}]$$

Nevertheless, when I have to calculate $E[\hat{\mu_x}^2]$ (I suposse it's this way):

$$E[\hat{\mu_x}^2] = \dfrac{1}{N^2} E\left[ \left( \sum_{n = 0}^{N-1}x(n) \right)^2 \right]$$

I don't know hot to do it. Moreover, I don't know if $E[x(n) \hat{\mu_x}] = E[x(n)] E[\hat{\mu_x}]$ (I think that's not correct in general), and I have tried to solve it in another way, but anything comes to my mind.

I hope someone can help me. Thank you for your responses.


The complexity of that demonstration gets reduced if you don’t start by expanding the square parenthesis. That is because at the end you will need to have somehow the expression of the variance of your random process and with the method you are proposing there is no possible way of getting that expression. The variance of a random process is:


This is how I would do it:

First of all, you start by substracting the mean of the random process to both members of the square parenthesis:

$\widehat{\sigma^2_{x}}=\dfrac{1}{N}\sum_{n=0}^{N-1} {[(x(n)-\mu_{x})-(\widehat{\mu_{x}}-\mu_{x})]^2}$

Now you cand expand the square expression, obtaining:

$\widehat{\sigma^2_{x}}=\dfrac{1}{N}\sum_{n=0}^{N-1} {(x(n)-\mu_{x})^2-2(x(n)-\mu_{x})(\widehat{\mu_{x}}-\mu_{x})+(\widehat{\mu_{x}}-\mu_{x})^2]}$

If you observe the first and the last term inside the summatory, the first one gives you directly the variance of the random process and the last one is the variance of the mean estimator.

Replacing the last term by its value (which is $\dfrac{\sigma^2_{x}}{N}$) and taking average in the whole expression you will get something like this:

$E[\widehat{\sigma^2_{x}}]=\sigma^2_{x}+\dfrac{1}{N}\sigma^2_{x}-\dfrac{2}{N}\sum_{n=0}^{N-1} {E[(x(n)-\mu_{x})(\widehat{\mu_{x}}-\mu_{x})]]}$

Note that since you are replacing $\sum_{n=0}^{N-1} {(x(n)-\mu_{x})}^2$ by $\sigma^2_{x}$ you are taking away that term of the summatory, which implies that $\sigma^2_{x}$ is multiplied by N (length of the summatory):

$E[\widehat{\sigma^2_{x}}]=\dfrac{1}{N}\sigma^2_{x}N+\dfrac{1}{N}\sigma^2_{x}\dfrac{1}{N}N-\dfrac{2}{N}\sum_{n=0}^{N-1} {E[(x(n)-\mu_{x})(\widehat{\mu_{x}}-\mu_{x})]]}$

The last step is to make a connection between the term inside the summatory and the variance of the random process. This is simple if you replace the value of the sample mean by the one you have already written above.


Now we can only focus on that expression:

$\sum_{n=0}^{N-1} {E[(x(n)-\mu_{x})(\dfrac{1}{N}\sum_{n=0}^{N-1}{x(n)-\mu_{x})]}}$

Considering that each element of $x(n)$ is uncorrelated, we can separate the mean in two terms:


I am also considering that $\widehat{\mu_{x}}$ is refered to the same x(n). This is why you can separate the mean as I said. If it wasn’t so, and the two variables are uncorrelated, this term will be 0:


Thus, operating with the second mean gets you to:


As you can see, at this point you have an expression that looks like the variance of the random process.

Returning a few steps, you will realize that your other term of the summatory is also $E[x(n)-\mu_{x}]$ , which leads us to:

$\dfrac{2}{N}\sum_{n=0}^{N-1} {E[(x(n)-\mu_{x})(\widehat{\mu_{x}}-\mu_{x})}]=\dfrac{2}{N^2}\sum_{n=0}^{N-1}{E[(x(n)-\mu_{x})^2]}$

Writing again the expression of the mean of the sample variance:

$\sigma^2_{x}+\dfrac{1}{N}\sigma^2_{x}-\dfrac{2}{N^2}\sum_{n=0}^{N-1}{E[(x(n)-\mu_{x})^2]} => \sigma^2_{x}+\dfrac{1}{N}\sigma^2_{x}-\dfrac{2}{N}\sigma^2_{x}$

Finally, once you have the variance at each term, you reach the end of the demonstration.


Hope it helped.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.