This might be a very naive question.

Wikipedia describes an exponential family as a distribution

$f(x \mid \theta) = h(x)\exp( - \theta x - A(\theta))$

where $A(\theta) = \int h(x)\exp( - \theta x) \, dx$

I want to understand the role of the underlying measure or basically the $h(x)$ in the above equation.

For instance can any function of $\theta$ be written as $A(\theta)$ for some choice of $h(x)$.

  • $\begingroup$ An exponential family is not a distribution, but a set of distributions. What you've written is a density. You could say that $h(x)\,dx$ is the measure, and $dx$ is itself a measure --- the one that assigns to every interval its length. ${}\qquad{}$ $\endgroup$ Oct 20, 2015 at 16:32

1 Answer 1


Every probability density is with respect to a measure. The Lebesgue measure is the measure that assigns to every interval its length, so that the measure of the interval $(2,6) = \{ x : 2<x<6\}$ is $6-2=4$. The standard normal density function is $\displaystyle x\mapsto\frac 1 {\sqrt{2\pi}} e^{-x^2/2}$ and that is a density with respect to Lebesgue measure. In integrals $\text{“}dx\text{''}$ represents the Lebesgue measure, and that means $$ \int_{(2,6)} 13\,dx = \Big( 13\times\text{the measure of } (2,6) \Big) = 13\times 4 = 52. $$ If one multiplies the density $\displaystyle \frac 1 {\sqrt{2\pi}} e^{-x^2/2}$ with respect to Lebesgue measure by the Lebesgue measure $dx$ then one gets a measure called $\displaystyle \overbrace{\frac 1 {\sqrt{2\pi}} e^{-x^2/2} \, dx}^\text{This includes “$dx$''.}$, which is the standard normal distribution. The measure that that assigns to the interval $(2,6)$ is $$ \int_{(2,6)} \frac 1 {\sqrt{2\pi}} e^{-x^2/2} \, dx = \int_2^6 \frac 1 {\sqrt{2\pi}} e^{-x^2/2} \, dx, $$ the probability that a random variable with a standard normal distribution is in the interval $(2,6)$.

The measure considered in the definition of "exponential family" is not $h(x)$, but $h(x)\,dx$. And this is somewhat misleading, in that it suggests that there is always some density function $x\mapsto h(x)$ that gets integrated with respect to Lebesgue measure to get the measure $h(x)\,dx$. But in fact one often considers discrete measures, made up entirely of point masses, that assign positive measure only to sets of Lebesgue measure zero, so there is no density that can be integrated with respect to Lebesgue measure to obtain that discrete measure. For example \begin{align} x\mapsto {} & \binom n x p^x (1-p)^{n-x} = \binom n x \left( \frac p {1-p} \right)^x (1-p)^n \\[12pt] = {} & h(x) \exp\left( -x\theta - A(\theta) \right) \\ & \text{where }\theta = -\log \frac p {1-p} \text{ and }A(\theta) = -n\log(1-p) \text{ and }h(x) = \binom n x. \end{align} Here one integrates not with respect to Lebesgue measure but with respect to counting measure on the set $\{0,1,2,\ldots,n\}$, so that the measure of any set $A$ is the number of points in $A\cap\{0,\ldots,n\}$, and so the integral is a sum: $$ \int g(x) \, dc(x) = \sum_{x=0}^n g(x) \text{ where $c$ is the counting measure on } \{0,\ldots,n\}. $$

As to which functions can be realized as $\theta\mapsto A(\theta)$, that is a matter of inverting a Laplace transform. There are some inversion theorems, but I'm not sure if we need separate inversion theorems for different measures.

  • $\begingroup$ Thanks for the detailed answer. It clarified a lot of things for me. Can you give me a reference for the last part of your statement the inversion of Laplace transforms? $\endgroup$ Nov 3, 2015 at 22:11
  • $\begingroup$ @user1189053 : I don't have such a reference at the tip of my tongue, but here is my rationale. The integral of any probability density must be $1$: $$\int_\Theta h(x)\exp( - \theta x - A(\theta)) \, dx = 1.$$ Hence $$e^{A(\theta)} = \int_\Theta h(x) \exp(-\theta x)\,dx.$$ That says $\theta \mapsto e^{A(\theta)}$ is a Laplace transform of $h$. ${}\qquad{}$ $\endgroup$ Nov 3, 2015 at 22:23
  • $\begingroup$ I agree with that. What I wanted to know was that whether the transform is always invertible. $\endgroup$ Nov 4, 2015 at 15:17
  • $\begingroup$ I just came across this answer now and just wanted to say that it's incredible. Cuts right through the sea of muddy explanations of the definition of an exponential family. +1. $\endgroup$
    – gogurt
    Jan 9, 2018 at 0:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .