1
$\begingroup$

We have the exponential family: $$ f_\mathbf{X}(x;\theta) = h(x)\exp\{\langle\theta, T(x)\rangle-A(\theta)\} $$ where the parameter vector $\theta$ is often referred to as canonical parameter or natural parameter.

Not hard to prove: $\frac{\partial}{\partial \theta}A(\theta) = \mathbb E[T(X)]$

Suppose that the exponential family is minimal. Define $\mu = \mathbb E[T(X)]$, we will get another parameterization - $\mu$ mean parameterization - of the exponential family.

Let $X=\{X_1,X_2,\cdots,X_n\}$, by maximum likelihood estimation, we have: $$ L(\theta;X) =\prod_{i=1}^N f(x_i;\theta) = \prod_{i=1}^N h(x_i)\exp\{\langle\eta(\theta), T(x_i)\rangle-A(\theta)\} $$ the score function is: $$ \begin{align*} S(\theta;X)&=\frac{\partial}{\partial \theta}\log L(\theta;X)\\ &= \frac{\partial}{\partial \theta} \left\{ \log\left(\prod_{i=1}^N h(x_i)\right) + \theta^T\left( \sum_{i=1}^N T(x_i) \right) - NA(\theta) \right\} \\ &= \sum_{i=1}^N T(x_i) - N\frac{\partial}{\partial \theta} A(\theta) \end{align*} $$ The maximum likelihood estimator of $\theta$ is $\hat\theta_{MLE}=\arg_{\theta}\{S(\theta;X)=0\}$. When solving this equation, it's easy to prove $\hat\theta_{MLE}$ must satisfy the following equation: $$ \frac{\partial}{\partial \theta} A(\theta) = \frac1N\sum_{i=1}^N T(x_i) $$ The right hand side of the above equation is the expectation of the sufficient statistic $T(X)$, i.e. $\mathbb E[T(X)]$

We defined before $\mu := \mathbb E[T(X)]$, therefore, we have the following equation: $$ \hat\mu_{MLE}:=\mathbb E[T(X)]=\frac1N\sum_{i=1}^N T(x_i)=\frac{\partial}{\partial\theta} A(\hat\theta_{MLE}) $$

$\hat\mu_{MLE}$ is unbiased because $$\mathbb E [\hat\mu_{MLE}] = \frac1N\sum_{i=1}^N\mathbb E[T(X_i)] = \frac1N N\mu = \mu$$

Then I calculated the Fisher Information of exponential family w.r.t. canonical parameter $\theta$ $$ \begin{align*} I(\theta) &= -\mathbb E\left[\frac{\partial^2}{\partial\theta^2}\log L(\theta; X)\right] \\ &= -\mathbb E\left[\frac{\partial^2}{\partial\theta^2}\log f_X(x;\theta)\right] \\ &= -\mathbb E\left[-\frac{\partial^2}{\partial\theta^2}A(\theta)\right] \\ &= \mathbb E[Var[T(X)]] \\ &= Var[T(X)] \end{align*} $$

I wonder:

  1. If I'm correct, and

  2. How can I work out the fisher information $I(\hat\mu_{MLE})$ w.r.t. the mean parameterization estimator $\hat\mu_{MLE}$ to prove that $\hat\mu_{MLE}$ is a minimum-variance unbiased estimator?

My reference is: The Exponential Family: Basics, M. Jordan, 2009. I'm relly confused when reading the relevant part in that chapter.

I would be grateful if you could answer my question. Thanks!

BTW, the link The Exponential Family: Basics, M. Jordan, 2009. only provide the Chapter 8. Does anyone know the name of the book? I haven't find it anywhere.

$\endgroup$
2
  • $\begingroup$ I'm not sure why you write "$\hat\mu_{MLE} = \mathbb E[T(X)] =\cdots$". $\endgroup$ Jul 11, 2023 at 7:41
  • $\begingroup$ Hey, thank you for your time. I've updated my post. I hope I've made myself clear. $\endgroup$
    – Jason Ye
    Jul 11, 2023 at 9:27

1 Answer 1

0
$\begingroup$

If you estimate $\psi(\theta)$ then Cramer Roa Lower bound for an unbiased estimator $\hat{\psi}(\theta)$ is, $$Var(\hat{\psi}(\theta)) \geq \nabla_{\theta} \psi(\theta)^T I(\theta)^{-1} \nabla_{\theta} \psi(\theta)$$

See theorem 3 in https://www.ece.iastate.edu/~namrata/EE527_Spring08/l2.pdf OR Appendix A in https://www.ele.uri.edu/faculty/kay/New%20web/downloadable%20files/Nageha_complex%20CRLB.pdf

We are going to apply the above formula in the below:

For $X_1,....,X_n$,

$$\prod_{i=1}^n f(x_i |\theta) = \prod_{i=1}^n h(x_i) \times exp(\langle \theta, \sum_{i=1}^n T(x_i) \rangle - nA(\theta))$$

$$g(x_1,...,x_n,\theta) = \log \left(\prod_{i=1}^n f_(x_i | \theta) \right) = \sum_{i=1}^n h(x_i) + \langle \theta, \sum_{i=1}^n T(x_i) \rangle - nA(\theta)$$

$$\nabla_{\theta} g(x_1,...,x_n,\theta) = \sum_{i=1}^n T(x_i) - n\nabla_{\theta} A(\theta)$$

$$[I(\theta)]_{ij} = E\left(-\frac{\partial^2 g(x_1,...,x_n,\theta) }{\partial \theta_i \theta_j} \right) = n \frac{\partial^2 A(\theta)}{\partial \theta_i \theta_j} = n^2 Cov \left( \left[\frac{1}{n} \sum_k T(x_k) \right]_i, \left[\frac{1}{n}\sum_k T(x_k) \right]_j \right)$$

We have, $$[\mu(\theta)]_i = E([T(x)]_i) = \frac{\partial A(\theta)}{\partial \theta_i}$$ $$\nabla_{\theta} [\mu(\theta)]_i^T I(\theta)^{-1} \nabla_{\theta} [\mu(\theta)]_i = \left( \left[\frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_j} : j \right] \right)^T I(\theta)^{-1} \left(\left[\frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_j} : j \right] \right) = \left(\left[\frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_j} : j \right] \right)^T \left(n\left[\frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_j} : i,j \right] \right)^{-1} \left(\left[\frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_j} : j \right] \right) = \frac{1}{n} \frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_i} = n Cov \left( \left[\frac{1}{n} \sum_k T(x_k) \right]_i, \left[\frac{1}{n}\sum_k T(x_k) \right]_i \right) = \frac{Var([T(x)]_i)}{n}$$

$Var([\mu_{MLE}]_i) = Var(\left[\frac{1}{n} \sum_{k=1}^n T(x_k)\right]_i) = \frac{Var([T(X)]_i)}{n}$

So $Var([\mu_{MLE}]_i) = \nabla_{\theta} [\mu(\theta)]_i^T I(\theta)^{-1} \nabla_{\theta} [\mu(\theta)]_i $

So your estimator achieves Cramer Roa Lower bound. Hence Minimum Variance unbiased estimator.

$\endgroup$
3
  • $\begingroup$ If you are convinced, accept the answer by clicking to the left of my answer. $\endgroup$
    – Balaji sb
    Jul 11, 2023 at 9:44
  • 1
    $\begingroup$ I would love to, but I am new here and do not have enough reputation to cast a vote. I've pressed useful, and my feedback has been recorded by the system (but not shown). $\endgroup$
    – Jason Ye
    Jul 11, 2023 at 10:29
  • $\begingroup$ That's alright. Cheers ! $\endgroup$
    – Balaji sb
    Jul 11, 2023 at 10:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .