Calculate the Variance of Frequency Estimation

Suppose we have total $$n$$ samples of $$(X,Y)$$, where $$(X,Y)$$ is discrete variable. Now I want to estimate the conditional probability $$P(Y=y\mid X=x)$$ . The direct idea is to estimate the probability using frequency, namely, $$\hat{P}\left( Y=y\mid X=x \right) =\frac{\sum_{i=1}^n{\mathbb{I} \left( Y_i=y,X_i=x \right)}}{\sum_{i=1}^n{\mathbb{I} \left( X_i=x \right)}}$$ where $$\mathbb{I}$$ is the indicative function.

My question is how to calculate the variance of this estimate $$\mathrm{Var}(\hat{P}\left( Y=y\mid X=x \right))$$. Furthermore, how can we calculate the covariance of frequency estimation for different categories, $$\mathrm{Cov(}\hat{P}\left( Y=y\mid X=x \right) ,\hat{P}\left( Y=y'\mid X=x' \right) )$$ ?

I am trying to solve this problem by first solving a relatively simple problem and seeing if there is a way of thinking. For $$\hat{P}\left( Y=y\right) =\frac{\sum_{i=1}^n{\mathbb{I} \left( Y_i=y \right)}}{n}$$，we can easily calculate his variance \begin{aligned} \mathrm{Var}\left( \hat{P}\left( Y=y \right) \right) &=\mathrm{Var}\left( \frac{\sum_{i=1}^n{\mathbb{I} \left( Y_i=y \right)}}{n} \right)\\ &=\frac{1}{n^2}\mathrm{Var}\left( \sum_{i=1}^n{\mathbb{I} \left( Y_i=y \right)} \right)\\ &=\frac{1}{n^2}nP\left( Y=y \right) \left( 1-P\left( Y=y \right) \right)\\ &=\frac{P\left( Y=y \right) \left( 1-P\left( Y=y \right) \right)}{n}\\ \end{aligned} The third equation can be regarded as the variance of binomial distribution. However, when I tried to extrapolate this method to the estimation of conditional probability, I got stuck. The numerator and denominator of conditional frequency both have random variables, which I cannot solve.

How can I calculate the variance of this estimate? Or could you provide a better way to solve the problem. Thank you in advance.

• You can't calculate the variance of that estimator without conditions on $X_i$. If no $X_i$ are equal to $x$ then it isn't even defined. If you calculate the variance conditional on at leat one $X_i=x$, you can use the fact that you know the variance conditional on given realisations of $X_i$. You can find the first and second moments of the estimator conditional on given $X_I$, the sum over all possibilities of $X_i$ to find the unconditional moments (except being conditional on some $X_i=x$). Commented Apr 16, 2023 at 3:02
• @ZoeAllen It is possible that my question is not clearly stated, $(X_i,Y_i)$ is the sample collected from the $(X,Y)$ distribution.
– 叶心萤
Commented Apr 16, 2023 at 3:14

\begin{aligned} \mathrm{Var}\left( \hat{P} \right) &=E\left( \mathrm{Var}\left( \hat{P}\mid \sum_{i=1}^n{I\left( X_i=x \right)} \right) \right) +\mathrm{Var}\left( E\left( \hat{P}\mid \sum_{i=1}^n{I\left( X_i=x \right)} \right) \right)\\ &=E\left( \frac{1}{\left( \sum_{i=1}^n{I\left( X_i=x \right)} \right) ^2}\mathrm{Var}\left( \sum_{i=1}^n{I\left( X_i=x,Y_i=y \right)}\mid \sum_{i=1}^n{I\left( X_i=x \right)} \right) \right) +\mathrm{Var}\left( \frac{1}{\sum_{i=1}^n{I\left( X_i=x \right)}}E\left( P\sum_{i=1}^n{I\left( X_i=x,Y_i=y \right)}\mid \sum_{i=1}^n{I\left( X_i=x \right)} \right) \right)\\ &=E\left( \frac{\sum_{i=1}^n{I\left( X_i=x \right)}\left( P\left( Y\mid X \right) \right) \left( 1-P\left( Y\mid X \right) \right)}{\left( \sum_{i=1}^n{I\left( X_i=x \right)} \right) ^2} \right) +\mathrm{Var}\left( \frac{\sum_{i=1}^n{I\left( X_i=x \right)}P\left( Y\mid X \right)}{\sum_{i=1}^n{I\left( X_i=x \right)}} \right)\\ &=E\left( \frac{P\left( Y\mid X \right) \left( 1-P\left( Y\mid X \right) \right)}{\sum_{i=1}^n{I\left( X_i=x \right)}} \right) =\frac{P\left( Y\mid X \right) \left( 1-P\left( Y\mid X \right) \right)}{nP\left( X \right)}\\ \end{aligned}