# How to calculate Fisher Information of exponential family w.r.t. mean parameterization in maximum likelihood estimation?

We have the exponential family: $$f_\mathbf{X}(x;\theta) = h(x)\exp\{\langle\theta, T(x)\rangle-A(\theta)\}$$ where the parameter vector $$\theta$$ is often referred to as canonical parameter or natural parameter.

Not hard to prove: $$\frac{\partial}{\partial \theta}A(\theta) = \mathbb E[T(X)]$$

Suppose that the exponential family is minimal. Define $$\mu = \mathbb E[T(X)]$$, we will get another parameterization - $$\mu$$ mean parameterization - of the exponential family.

Let $$X=\{X_1,X_2,\cdots,X_n\}$$, by maximum likelihood estimation, we have: $$L(\theta;X) =\prod_{i=1}^N f(x_i;\theta) = \prod_{i=1}^N h(x_i)\exp\{\langle\eta(\theta), T(x_i)\rangle-A(\theta)\}$$ the score function is: \begin{align*} S(\theta;X)&=\frac{\partial}{\partial \theta}\log L(\theta;X)\\ &= \frac{\partial}{\partial \theta} \left\{ \log\left(\prod_{i=1}^N h(x_i)\right) + \theta^T\left( \sum_{i=1}^N T(x_i) \right) - NA(\theta) \right\} \\ &= \sum_{i=1}^N T(x_i) - N\frac{\partial}{\partial \theta} A(\theta) \end{align*} The maximum likelihood estimator of $$\theta$$ is $$\hat\theta_{MLE}=\arg_{\theta}\{S(\theta;X)=0\}$$. When solving this equation, it's easy to prove $$\hat\theta_{MLE}$$ must satisfy the following equation: $$\frac{\partial}{\partial \theta} A(\theta) = \frac1N\sum_{i=1}^N T(x_i)$$ The right hand side of the above equation is the expectation of the sufficient statistic $$T(X)$$, i.e. $$\mathbb E[T(X)]$$

We defined before $$\mu := \mathbb E[T(X)]$$, therefore, we have the following equation: $$\hat\mu_{MLE}:=\mathbb E[T(X)]=\frac1N\sum_{i=1}^N T(x_i)=\frac{\partial}{\partial\theta} A(\hat\theta_{MLE})$$

$$\hat\mu_{MLE}$$ is unbiased because $$\mathbb E [\hat\mu_{MLE}] = \frac1N\sum_{i=1}^N\mathbb E[T(X_i)] = \frac1N N\mu = \mu$$

Then I calculated the Fisher Information of exponential family w.r.t. canonical parameter $$\theta$$ \begin{align*} I(\theta) &= -\mathbb E\left[\frac{\partial^2}{\partial\theta^2}\log L(\theta; X)\right] \\ &= -\mathbb E\left[\frac{\partial^2}{\partial\theta^2}\log f_X(x;\theta)\right] \\ &= -\mathbb E\left[-\frac{\partial^2}{\partial\theta^2}A(\theta)\right] \\ &= \mathbb E[Var[T(X)]] \\ &= Var[T(X)] \end{align*}

I wonder:

1. If I'm correct, and

2. How can I work out the fisher information $$I(\hat\mu_{MLE})$$ w.r.t. the mean parameterization estimator $$\hat\mu_{MLE}$$ to prove that $$\hat\mu_{MLE}$$ is a minimum-variance unbiased estimator?

My reference is: The Exponential Family: Basics, M. Jordan, 2009. I'm relly confused when reading the relevant part in that chapter.

I would be grateful if you could answer my question. Thanks!

BTW, the link The Exponential Family: Basics, M. Jordan, 2009. only provide the Chapter 8. Does anyone know the name of the book? I haven't find it anywhere.

• I'm not sure why you write "$\hat\mu_{MLE} = \mathbb E[T(X)] =\cdots$". Commented Jul 11, 2023 at 7:41
• Hey, thank you for your time. I've updated my post. I hope I've made myself clear. Commented Jul 11, 2023 at 9:27

## 1 Answer

If you estimate $$\psi(\theta)$$ then Cramer Roa Lower bound for an unbiased estimator $$\hat{\psi}(\theta)$$ is, $$Var(\hat{\psi}(\theta)) \geq \nabla_{\theta} \psi(\theta)^T I(\theta)^{-1} \nabla_{\theta} \psi(\theta)$$

We are going to apply the above formula in the below:

For $$X_1,....,X_n$$,

$$\prod_{i=1}^n f(x_i |\theta) = \prod_{i=1}^n h(x_i) \times exp(\langle \theta, \sum_{i=1}^n T(x_i) \rangle - nA(\theta))$$

$$g(x_1,...,x_n,\theta) = \log \left(\prod_{i=1}^n f_(x_i | \theta) \right) = \sum_{i=1}^n h(x_i) + \langle \theta, \sum_{i=1}^n T(x_i) \rangle - nA(\theta)$$

$$\nabla_{\theta} g(x_1,...,x_n,\theta) = \sum_{i=1}^n T(x_i) - n\nabla_{\theta} A(\theta)$$

$$[I(\theta)]_{ij} = E\left(-\frac{\partial^2 g(x_1,...,x_n,\theta) }{\partial \theta_i \theta_j} \right) = n \frac{\partial^2 A(\theta)}{\partial \theta_i \theta_j} = n^2 Cov \left( \left[\frac{1}{n} \sum_k T(x_k) \right]_i, \left[\frac{1}{n}\sum_k T(x_k) \right]_j \right)$$

We have, $$[\mu(\theta)]_i = E([T(x)]_i) = \frac{\partial A(\theta)}{\partial \theta_i}$$ $$\nabla_{\theta} [\mu(\theta)]_i^T I(\theta)^{-1} \nabla_{\theta} [\mu(\theta)]_i = \left( \left[\frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_j} : j \right] \right)^T I(\theta)^{-1} \left(\left[\frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_j} : j \right] \right) = \left(\left[\frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_j} : j \right] \right)^T \left(n\left[\frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_j} : i,j \right] \right)^{-1} \left(\left[\frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_j} : j \right] \right) = \frac{1}{n} \frac{\partial^2 A(\theta)}{\partial \theta_i \partial \theta_i} = n Cov \left( \left[\frac{1}{n} \sum_k T(x_k) \right]_i, \left[\frac{1}{n}\sum_k T(x_k) \right]_i \right) = \frac{Var([T(x)]_i)}{n}$$

$$Var([\mu_{MLE}]_i) = Var(\left[\frac{1}{n} \sum_{k=1}^n T(x_k)\right]_i) = \frac{Var([T(X)]_i)}{n}$$

So $$Var([\mu_{MLE}]_i) = \nabla_{\theta} [\mu(\theta)]_i^T I(\theta)^{-1} \nabla_{\theta} [\mu(\theta)]_i$$

So your estimator achieves Cramer Roa Lower bound. Hence Minimum Variance unbiased estimator.

• If you are convinced, accept the answer by clicking to the left of my answer. Commented Jul 11, 2023 at 9:44
• I would love to, but I am new here and do not have enough reputation to cast a vote. I've pressed useful, and my feedback has been recorded by the system (but not shown). Commented Jul 11, 2023 at 10:29
• That's alright. Cheers ! Commented Jul 11, 2023 at 10:33