Maximum a posteriori estimates can be approached by sampling from an 'exponentiated' version of the posterior. That is, samples from $p_\eta(\theta \mid x) \propto p(\theta \mid x)^\eta$ approach the MAP estimate as $\eta\to\infty$. I was wondering what would happen if you introduced this $\eta\ge1$ in the following Dirichlet-categorical model: \begin{align} (\theta_1, \dots, \theta_K) &\sim \text{Dirichlet}(\alpha_1, \dots, \alpha_K) \\ x &\sim \text{Categorical}(\hat{\theta}_1, \dots, \hat{\theta}_K), \quad \text{where } \hat{\theta}_k = \frac{\theta_k^\eta}{\sum_{j=1}^K \theta_j^\eta}. \end{align} I am mainly interested in the posterior $p(\boldsymbol\theta \mid x)$. For $\eta=1$ this is a Dirichlet with parameters $\boldsymbol\alpha + \boldsymbol c$ if $\boldsymbol c$ is the vector of counts; i.e. $c_k = \mathbb{1}(x = k)$. But for $\eta>1$ I think it is not a Dirichlet, because the normalizing 'constant' $\bigl(\sum_j \theta_j^\eta\bigr)^{-1}$ of the categorical depends on $\boldsymbol \theta$:

$$ p(\boldsymbol \theta \mid x) \propto p_\eta(x \mid \boldsymbol\theta) \cdot p(\boldsymbol\theta) \propto \frac{\theta_x^\eta}{\sum_{j=1}^K \theta_j^\eta} \cdot \prod_{k=1}^K \theta_k^{\alpha_k - 1} = \Bigl(\sum_{j=1}^K \theta_j^\eta\Bigr)^{-1} \cdot \prod_{k=1}^K \theta_k^{\alpha_k + \eta c_k - 1}. $$

But what is it, if anything? Concretely, I have two questions:

  1. Is this a know distribution?
  2. If not, is it possible to compute the normalising constant $\int_\Delta \bigl(\sum_j \theta_j^\eta\bigr)^{-1} \prod_k \theta^{\alpha_k + \eta c_k - 1} d\boldsymbol\theta$, with $\Delta$ being the simplex? I have no idea if there is anything you can do with the sum and all suggestions (including 'probably not') would be very much appreciated.


(I believe it is also kind of intuitive that the posterior is not Dirichlet. As $\eta \to \infty$ you tend to get $x = \text{argmax}_k \theta_k$. After observing $x$ there is only one part of the simplex where $\boldsymbol\theta$ could have lain: $S_x=\{\boldsymbol\theta \in \Delta: \theta_x \ge \theta_k, \forall k\}$. So the posterior would be $p(\boldsymbol\theta \mid x) = p(\boldsymbol\theta \mid \boldsymbol\theta \in S_x)$, which looks like the prior restricted to $S_x$. That sounds pretty discontinuous and un-Dirichlet-ish to me — right?)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.