I know that $E[x] = p$ for some Bernoulli random variable X with parameter $p$ where $P(X = 1) = p$ and $P(X = 0) = 1 − p$. Similarly $E[X^2] = p$ and $Var[X] = p − p^2 = p(1 − p)$.

However, I'm having trouble wrapping my head around the following. Let's say $X_i ∼ Bernoulli(p)$. I was told that we can determine the value of $p$ by using some number $t$ of i.i.d. samples $X_1, ... , X_t ∼ Bernoulli(p)$. What is the expected value of the estimate?

  1. My intuition isn't quite clear here. My initial attempt it is, let us say we have a coin that lands on heads with $p$ and tails with $1-p$. Then, we can flip it some $t$ times, and if we do something like $\frac{\text{# of heads}}{\text{total # of slips}}$ or equivalently $X = \frac{1}{t}\sum^{t}_{i=1}X_i$, then we can estimate $p$. And as $t$ gets larger, the stronger our estimate will be. Is this the right intuition here?

Secondly, if my intuition is correct, then what exactly is the expected value and variance of $X$ in terms of $p$ and $t$. I realize this might be an obvious question, but I'm having trouble differentiating it from just $E[x] = p$.

Am I just suppose to be doing $E[X] = E[X_1] + E[X_2] ...$? How do I put it in terms of $p$ and $t$.

  • 1
    $\begingroup$ $X$ will have the same expected value as each $X_i$ due to linearity of the expected value. Additionally, the variance of $X$ will be the variance of $X_i$ divided by $t$. So you can see that adding more samples will reduce the variance of the estimate. $\endgroup$ Oct 5 '20 at 22:57

You apply the Linearity of Expectation.

Since the mean estimator is indeed $\bar X=\tfrac {\sum_{i=1}^tX_t}t$ then the expectation is:-

$$\begin{align}\mathsf E(\bar X) &= \mathsf E\left(\tfrac 1 t\sum_{i=1}^t X_i\right)\\[1ex]&=\tfrac 1t\sum_{i=1}^t\mathsf E(X_i)\end{align}$$

The rest is that these $t$ samples are identically distributed, and for all $t$ samples their expectation is $p$.

So, yeah, $\mathsf E(\bar X)=p$

Similarly for the variance of the mean estimator, we also use independence (and so the variance of the sum equals the sum of variance):- $$\begin{align}\mathsf{Var}(\bar X)&=\mathsf {Var}(\tfrac 1t\sum_{i=1}^t X_i)\\[1ex]&=\sum_{i=1}^t\mathsf{Var}(\tfrac 1{t}X_i)\\[1ex]&~~\vdots\end{align}$$

Recall we have for any scalar $a$ and random variable $Z$, that: $\mathsf {Var}(aZ)=a^2\mathsf{Var}(Z)$

  • 1
    $\begingroup$ So for $Var(X)$, I get $\frac{1}{t^2}\sum^{t}_{i=1}{Var(X_i)}$. And we have $Var(X_i) = E({X_i}^2) - E(X_i)^2 = E({X_i}^2) - p^2$. This is where I'm stuck. From an intuitive sense what does it mean to say $E({X_i}^2)$? Expectation of getting $X_i$ AND $X_i$ ? Not sure how to simply that. $\endgroup$
    – Jonathan
    Oct 6 '20 at 15:59
  • 1
    $\begingroup$ I know the answer for $E({X_i}^2)$ is $p$. And so the final answer is $\mathsf{Var}(\bar X) = \frac{p(1-p)}{t}$. But unable to intuitively wrap my head around why we are using this value. $\endgroup$
    – Jonathan
    Oct 6 '20 at 16:05
  • $\begingroup$ @Jonathan , You see that $X_i^2$ takes on values entirely determined by the value for $X_i$, so by definition: $$\begin{align}\mathsf E(X_i^2)&=0^2\cdot\mathsf P(X_i=0)+1^2\cdot\mathsf P(X_i=1)\\ &=0+p\end{align}$$ $\endgroup$ Oct 6 '20 at 23:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.