Expected value of a the estimate of Bernoulli distribution

I know that $$E[x] = p$$ for some Bernoulli random variable X with parameter $$p$$ where $$P(X = 1) = p$$ and $$P(X = 0) = 1 − p$$. Similarly $$E[X^2] = p$$ and $$Var[X] = p − p^2 = p(1 − p)$$.

However, I'm having trouble wrapping my head around the following. Let's say $$X_i ∼ Bernoulli(p)$$. I was told that we can determine the value of $$p$$ by using some number $$t$$ of i.i.d. samples $$X_1, ... , X_t ∼ Bernoulli(p)$$. What is the expected value of the estimate?

1. My intuition isn't quite clear here. My initial attempt it is, let us say we have a coin that lands on heads with $$p$$ and tails with $$1-p$$. Then, we can flip it some $$t$$ times, and if we do something like $$\frac{\text{# of heads}}{\text{total # of slips}}$$ or equivalently $$X = \frac{1}{t}\sum^{t}_{i=1}X_i$$, then we can estimate $$p$$. And as $$t$$ gets larger, the stronger our estimate will be. Is this the right intuition here?

Secondly, if my intuition is correct, then what exactly is the expected value and variance of $$X$$ in terms of $$p$$ and $$t$$. I realize this might be an obvious question, but I'm having trouble differentiating it from just $$E[x] = p$$.

Am I just suppose to be doing $$E[X] = E[X_1] + E[X_2] ...$$? How do I put it in terms of $$p$$ and $$t$$.

• $X$ will have the same expected value as each $X_i$ due to linearity of the expected value. Additionally, the variance of $X$ will be the variance of $X_i$ divided by $t$. So you can see that adding more samples will reduce the variance of the estimate. Oct 5 '20 at 22:57

You apply the Linearity of Expectation.

Since the mean estimator is indeed $$\bar X=\tfrac {\sum_{i=1}^tX_t}t$$ then the expectation is:-

\begin{align}\mathsf E(\bar X) &= \mathsf E\left(\tfrac 1 t\sum_{i=1}^t X_i\right)\\[1ex]&=\tfrac 1t\sum_{i=1}^t\mathsf E(X_i)\end{align}

The rest is that these $$t$$ samples are identically distributed, and for all $$t$$ samples their expectation is $$p$$.

So, yeah, $$\mathsf E(\bar X)=p$$

Similarly for the variance of the mean estimator, we also use independence (and so the variance of the sum equals the sum of variance):- \begin{align}\mathsf{Var}(\bar X)&=\mathsf {Var}(\tfrac 1t\sum_{i=1}^t X_i)\\[1ex]&=\sum_{i=1}^t\mathsf{Var}(\tfrac 1{t}X_i)\\[1ex]&~~\vdots\end{align}

Recall we have for any scalar $$a$$ and random variable $$Z$$, that: $$\mathsf {Var}(aZ)=a^2\mathsf{Var}(Z)$$

• So for $Var(X)$, I get $\frac{1}{t^2}\sum^{t}_{i=1}{Var(X_i)}$. And we have $Var(X_i) = E({X_i}^2) - E(X_i)^2 = E({X_i}^2) - p^2$. This is where I'm stuck. From an intuitive sense what does it mean to say $E({X_i}^2)$? Expectation of getting $X_i$ AND $X_i$ ? Not sure how to simply that. Oct 6 '20 at 15:59
• I know the answer for $E({X_i}^2)$ is $p$. And so the final answer is $\mathsf{Var}(\bar X) = \frac{p(1-p)}{t}$. But unable to intuitively wrap my head around why we are using this value. Oct 6 '20 at 16:05
• @Jonathan , You see that $X_i^2$ takes on values entirely determined by the value for $X_i$, so by definition: \begin{align}\mathsf E(X_i^2)&=0^2\cdot\mathsf P(X_i=0)+1^2\cdot\mathsf P(X_i=1)\\ &=0+p\end{align} Oct 6 '20 at 23:07