1
$\begingroup$

While studying a slide deck, I encountered the following exercise:

We are to give a bias variance decomposition when the prediction is given as a probability distribution over $C$ classes. Let $P = [P_1, . . . , P_C ]$ be the ground truth class distribution associated to a particular input pattern. Assume the random estimator of class probabilities $$\bar{{P}} = [\bar{{P}}_1, . . . , \bar{{P}}_C ] $$ for the same input pattern. The error function is given by the KL-divergence between the ground truth and the estimated probability distribution: $$\text{Error} = E[D_{KL}(P||\bar{{P}})]$$

First, we would like to determine the mean of the class distribution estimator $\bar{{P}}$. We define the mean as the distribution that minimizes its expected KL divergence from the class distribution estimator, that is, the distribution $R$ that optimizes $$\begin{matrix}\min\\R \end{matrix} \space E[ D_{KL}(R||\bar{{P}})]$$

I have found a way to proof that this is:

$$ R = [R_1, . . . , R_C ] $$ $$ \text{where} \space\space R_i = \frac{{\exp\space E[\log \bar{P_i} ]}}{{\sum_j\exp\space E[\log \bar{P_j}]}} \space \space \space ∀ \space 1 ≤ i ≤ C.$$

We are now asked to proof that:

$$ Error(\hat{P}) = Bias(\hat{P}) + Var(\hat{P}) $$

where

$$Error(\hat{P}) = E[D_{KL}(P || \hat{P})$$

$$Bias(\hat{P}) = D_{KL}(P || R)$$ $$Var(\hat{P}) = E[D_{KL}(R || \hat{P})$$

I started by writing out the KL Divergence: $$Bias(\hat{P}) + Var(\hat{P}) = \sum_{i=1}^{C} \left (P_i \log \left (\frac{P_i}{R_i} \right ) \right) + E \left [ \sum_{i=1}^{c} \left (R_i \log \left (\frac{R_i}{\hat{P}_i} \right ) \right ) \right ]$$

But I don't know how to continue from here on.

$\endgroup$
1
  • $\begingroup$ Which slide deck is this from? $\endgroup$ – Ramanujan Dec 2 '19 at 10:17
2
$\begingroup$

$\newcommand{\E}{\mathbb{E}}$We want to proof following statement: $$Error(\hat{P}) = Bias(\hat{P}) + Var(\hat{P}) = \E[D_{KL}(P || \hat{P})].$$

We have \begin{align} Error(\hat{P}) & = \E[D_{KL}(P || \hat{P})] \\ & = \E\left[\sum_{i=1}^{N}P_{i} \log\left(\frac{P_{i}}{\hat{P_{i}}}\right)\right] \\ & = \E\left[\E\left[\log(P) - \log(\hat{P})\right]\right] \\ & = \E\left[\E\left[\log(P) - \log(R) + \log(R) - \log(\hat{P})\right]\right] \\ & = \E\left[\E\left[\log(P) - \log(R)\right] + \E\left[\log(R) - \log(\hat{P})\right]\right] \\ & = \E\left[\log(P) - \log(R)\right] + \E\left[\E\left[\log(R) - \log(\hat{P})\right]\right] \\ & = \sum_{i=1}^{N}P_{i} \log\left(\frac{P_{i}}{R_{i}}\right) + E\left[\sum_{i=1}^{N}R_{i} \log\left(\frac{R_{i}}{\hat{P_{i}}}\right)\right] \\ &= D_{KL}(P || R) + \E\left[D_{KL}(R || \hat{P})\right] \\ & = Bias(\hat{P}) + Var(\hat{P}) \end{align}

$\endgroup$
1
  • $\begingroup$ Do both $\mathbb{E}$ mean the same? $\endgroup$ – Ramanujan Dec 1 '19 at 23:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.