Bias Variance Decomposition for KL Divergence

Question

While studying a slide deck, I encountered the following exercise:

We are to give a bias variance decomposition when the prediction is given as a probability distribution over $C$ classes. Let $P = [P_1, . . . , P_C ]$ be the ground truth class distribution associated to a particular input pattern. Assume the random estimator of class probabilities $$\bar{{P}} = [\bar{{P}}_1, . . . , \bar{{P}}_C ] $$ for the same input pattern. The error function is given by the KL-divergence between the ground truth and the estimated probability distribution: $$\text{Error} = E[D_{KL}(P||\bar{{P}})]$$

First, we would like to determine the mean of the class distribution estimator $\bar{{P}}$. We define the mean as the distribution that minimizes its expected KL divergence from the class distribution estimator, that is, the distribution $R$ that optimizes $$\begin{matrix}\min\\R \end{matrix} \space E[ D_{KL}(R||\bar{{P}})]$$

I have found a way to proof that this is:

$$ R = [R_1, . . . , R_C ] $$ $$ \text{where} \space\space R_i = \frac{{\exp\space E[\log \bar{P_i} ]}}{{\sum_j\exp\space E[\log \bar{P_j}]}} \space \space \space ∀ \space 1 ≤ i ≤ C.$$

We are now asked to proof that:

$$ Error(\hat{P}) = Bias(\hat{P}) + Var(\hat{P}) $$

where

$$Error(\hat{P}) = E[D_{KL}(P || \hat{P})$$

$$Bias(\hat{P}) = D_{KL}(P || R)$$ $$Var(\hat{P}) = E[D_{KL}(R || \hat{P})$$

I started by writing out the KL Divergence: $$Bias(\hat{P}) + Var(\hat{P}) = \sum_{i=1}^{C} \left (P_i \log \left (\frac{P_i}{R_i} \right ) \right) + E \left [ \sum_{i=1}^{c} \left (R_i \log \left (\frac{R_i}{\hat{P}_i} \right ) \right ) \right ]$$

But I don't know how to continue from here on.

Answer 1

$\newcommand{\E}{\mathbb{E}}$We want to proof following statement: $$Error(\hat{P}) = Bias(\hat{P}) + Var(\hat{P}) = \E[D_{KL}(P || \hat{P})].$$

We have \begin{align} Error(\hat{P}) & = \E[D_{KL}(P || \hat{P})] \\ & = \E\left[\sum_{i=1}^{N}P_{i} \log\left(\frac{P_{i}}{\hat{P_{i}}}\right)\right] \\ & = \E\left[\E\left[\log(P) - \log(\hat{P})\right]\right] \\ & = \E\left[\E\left[\log(P) - \log(R) + \log(R) - \log(\hat{P})\right]\right] \\ & = \E\left[\E\left[\log(P) - \log(R)\right] + \E\left[\log(R) - \log(\hat{P})\right]\right] \\ & = \E\left[\log(P) - \log(R)\right] + \E\left[\E\left[\log(R) - \log(\hat{P})\right]\right] \\ & = \sum_{i=1}^{N}P_{i} \log\left(\frac{P_{i}}{R_{i}}\right) + E\left[\sum_{i=1}^{N}R_{i} \log\left(\frac{R_{i}}{\hat{P_{i}}}\right)\right] \\ &= D_{KL}(P || R) + \E\left[D_{KL}(R || \hat{P})\right] \\ & = Bias(\hat{P}) + Var(\hat{P}) \end{align}