Calculate the Variance of Frequency Estimation

Question

Suppose we have total $n$ samples of $(X,Y)$, where $(X,Y)$ is discrete variable. Now I want to estimate the conditional probability $P(Y=y\mid X=x)$ . The direct idea is to estimate the probability using frequency, namely, $$ \hat{P}\left( Y=y\mid X=x \right) =\frac{\sum_{i=1}^n{\mathbb{I} \left( Y_i=y,X_i=x \right)}}{\sum_{i=1}^n{\mathbb{I} \left( X_i=x \right)}} $$ where $\mathbb{I}$ is the indicative function.

My question is how to calculate the variance of this estimate $\mathrm{Var}(\hat{P}\left( Y=y\mid X=x \right))$. Furthermore, how can we calculate the covariance of frequency estimation for different categories, $\mathrm{Cov(}\hat{P}\left( Y=y\mid X=x \right) ,\hat{P}\left( Y=y'\mid X=x' \right) )$ ?

I am trying to solve this problem by first solving a relatively simple problem and seeing if there is a way of thinking. For $\hat{P}\left( Y=y\right) =\frac{\sum_{i=1}^n{\mathbb{I} \left( Y_i=y \right)}}{n}$，we can easily calculate his variance $$ \begin{aligned} \mathrm{Var}\left( \hat{P}\left( Y=y \right) \right) &=\mathrm{Var}\left( \frac{\sum_{i=1}^n{\mathbb{I} \left( Y_i=y \right)}}{n} \right)\\ &=\frac{1}{n^2}\mathrm{Var}\left( \sum_{i=1}^n{\mathbb{I} \left( Y_i=y \right)} \right)\\ &=\frac{1}{n^2}nP\left( Y=y \right) \left( 1-P\left( Y=y \right) \right)\\ &=\frac{P\left( Y=y \right) \left( 1-P\left( Y=y \right) \right)}{n}\\ \end{aligned} $$ The third equation can be regarded as the variance of binomial distribution. However, when I tried to extrapolate this method to the estimation of conditional probability, I got stuck. The numerator and denominator of conditional frequency both have random variables, which I cannot solve.

How can I calculate the variance of this estimate? Or could you provide a better way to solve the problem. Thank you in advance.

Answer 1

$$ \begin{aligned} \mathrm{Var}\left( \hat{P} \right) &=E\left( \mathrm{Var}\left( \hat{P}\mid \sum_{i=1}^n{I\left( X_i=x \right)} \right) \right) +\mathrm{Var}\left( E\left( \hat{P}\mid \sum_{i=1}^n{I\left( X_i=x \right)} \right) \right)\\ &=E\left( \frac{1}{\left( \sum_{i=1}^n{I\left( X_i=x \right)} \right) ^2}\mathrm{Var}\left( \sum_{i=1}^n{I\left( X_i=x,Y_i=y \right)}\mid \sum_{i=1}^n{I\left( X_i=x \right)} \right) \right) +\mathrm{Var}\left( \frac{1}{\sum_{i=1}^n{I\left( X_i=x \right)}}E\left( P\sum_{i=1}^n{I\left( X_i=x,Y_i=y \right)}\mid \sum_{i=1}^n{I\left( X_i=x \right)} \right) \right)\\ &=E\left( \frac{\sum_{i=1}^n{I\left( X_i=x \right)}\left( P\left( Y\mid X \right) \right) \left( 1-P\left( Y\mid X \right) \right)}{\left( \sum_{i=1}^n{I\left( X_i=x \right)} \right) ^2} \right) +\mathrm{Var}\left( \frac{\sum_{i=1}^n{I\left( X_i=x \right)}P\left( Y\mid X \right)}{\sum_{i=1}^n{I\left( X_i=x \right)}} \right)\\ &=E\left( \frac{P\left( Y\mid X \right) \left( 1-P\left( Y\mid X \right) \right)}{\sum_{i=1}^n{I\left( X_i=x \right)}} \right) =\frac{P\left( Y\mid X \right) \left( 1-P\left( Y\mid X \right) \right)}{nP\left( X \right)}\\ \end{aligned} $$