While studying a slide deck, I encountered the following exercise:
We are to give a bias variance decomposition when the prediction is given as a probability distribution over $C$ classes. Let $P = [P_1, . . . , P_C ]$ be the ground truth class distribution associated to a particular input pattern. Assume the random estimator of class probabilities $$\bar{{P}} = [\bar{{P}}_1, . . . , \bar{{P}}_C ] $$ for the same input pattern. The error function is given by the KL-divergence between the ground truth and the estimated probability distribution: $$\text{Error} = E[D_{KL}(P||\bar{{P}})]$$
First, we would like to determine the mean of the class distribution estimator $\bar{{P}}$. We define the mean as the distribution that minimizes its expected KL divergence from the class distribution estimator, that is, the distribution $R$ that optimizes $$\begin{matrix}\min\\R \end{matrix} \space E[ D_{KL}(R||\bar{{P}})]$$
I have found a way to proof that this is:
$$ R = [R_1, . . . , R_C ] $$ $$ \text{where} \space\space R_i = \frac{{\exp\space E[\log \bar{P_i} ]}}{{\sum_j\exp\space E[\log \bar{P_j}]}} \space \space \space ∀ \space 1 ≤ i ≤ C.$$
We are now asked to proof that:
$$ Error(\hat{P}) = Bias(\hat{P}) + Var(\hat{P}) $$
where
$$Error(\hat{P}) = E[D_{KL}(P || \hat{P})$$
$$Bias(\hat{P}) = D_{KL}(P || R)$$ $$Var(\hat{P}) = E[D_{KL}(R || \hat{P})$$
I started by writing out the KL Divergence: $$Bias(\hat{P}) + Var(\hat{P}) = \sum_{i=1}^{C} \left (P_i \log \left (\frac{P_i}{R_i} \right ) \right) + E \left [ \sum_{i=1}^{c} \left (R_i \log \left (\frac{R_i}{\hat{P}_i} \right ) \right ) \right ]$$
But I don't know how to continue from here on.