1
$\begingroup$

In a lecture this week my professor stated that

exponential families have convenient mathematical properties due to their natural parameterization such as the natural parameter space being convex.

Question: What does it mean that "the natural parameter space is convex"?


Some "thoughts": Does this suggest maximum likelihood estimators of the parameters always exist? What other mathematical properties of this result are useful?

$\endgroup$
5
  • 3
    $\begingroup$ This would probably make for a good discussion with your professor, showing interest in these remarks. However for the purpose of Math.SE content more context is needed to make it accessible to a broad range of Readers. You got a response limited to the issue of what it means for the parameter space to be convex, but the rest of the Question is pretty broad. Math.SE Questions should be more narrowly focused. You can get there by breaking up this material into well researched chunks, starting with what you understand well enough to check answers for. $\endgroup$
    – hardmath
    Feb 4, 2018 at 16:28
  • 2
    $\begingroup$ See this fairly recent Question at Cross Validated.SE about references for maximum likelihood estimation on exponential families. $\endgroup$
    – hardmath
    Feb 4, 2018 at 16:31
  • $\begingroup$ I will probably come back and expand my answer later today or tomorrow. $\endgroup$ Feb 4, 2018 at 18:09
  • $\begingroup$ This question only makes sense if one knows what the terms "exponential family" and "natural parameter space" mean, which are specialized terms typically known only by workers in mathematical statistics. $\endgroup$ Feb 5, 2018 at 3:28
  • 1
    $\begingroup$ This post has three separate parts: 1. "What does it mean that "the natural parameter space is convex"?" 2. "Does this suggest maximum likelihood estimators of the parameters always exist?" 3. "What other mathematical properties of this result are useful?" Now, 1. is utterly trivial (saying that the space of parameters is convex means that it is... well, convex) and it is the only part addressed in the accepted answer below (which basically states the parenthesis in this sentence). What happened to 2. and 3.? Is 1. on its own, really the kind of stuff questions on the site should be made of? $\endgroup$
    – Did
    Feb 5, 2018 at 10:59

2 Answers 2

0
$\begingroup$

That the natural parameter space is convex means that if $\alpha,\beta$ are two different points in the natural parameter space, then every point between $\alpha$ and $\beta$ is also within the natural parameter space.

$\endgroup$
0
$\begingroup$

Its a long time ago question. But ...

Let us consider the exponential family $$ f_\mathbf{X}(x;\theta) = h(x)\exp\{\langle\theta, T(x)\rangle-A(\theta)\} $$ where $$ A(\theta) = \log\left(\int_X h(x)\exp\{\langle\theta, T(x)\rangle\}~{\rm d}x\right) $$ is the log-partition function.

Now we rewrite the exponential family definition to: $$ f_\mathbf{X}(x;\theta) = \frac{1}{Z(\theta)}h(x)\exp\{\langle\theta, T(x)\rangle\} $$ Then we have partition function $Z(\theta)$: $$Z(\theta) = \int_X h(x)\exp\{\langle\theta, T(x)\rangle\}~{\rm d}x$$

The natural space is defined as: $$ \mathcal N = \left\{\theta:\int_X h(x)\exp\{\langle\theta, T(x)\rangle\}~{\rm d}x \lt \infty\right\} = \left\{\theta:Z(\theta) \lt \infty\right\} $$ Now we prove that the natural space is a convex set:

Consider two distinct parameter $\theta_1,\theta_2\in \mathcal N$ and $0\lt\lambda\lt1$. Let $\theta = \lambda\theta_1+(1-\lambda)\theta_2$ be a convex conbinition of $\theta_1$ and $\theta_2$. $$ \begin{aligned} Z(\theta) &= \exp\{A(\theta)\} = \exp\{A(\lambda\theta_1+(1-\lambda)\theta_2)\}\\ &=\int_X h(x)\exp\{\langle(\lambda\theta_1+(1-\lambda)\theta_2), T(x)\rangle\}~{\rm d}x \\ & = \int_X \left(h(x)^{\lambda}\exp\{\langle\lambda\theta_1, T(x)\rangle \}\right)\left(h(x)^{1-\lambda}\exp\{\langle(1-\lambda)\theta_2, T(x)\rangle\}\right)~{\rm d}x \\ &\leq \left(\int_X h(x)\exp\{\frac1\lambda\langle\lambda\theta_1, T(x)\rangle\} ~{\rm d}x \right)^\lambda \left(\int_X h(x)\exp\{\frac1{1-\lambda}\langle(1-\lambda)\theta_2, T(x)\rangle\} ~{\rm d}x \right)^{1-\lambda} \\ &=Z(\theta_1)^\lambda \cdot Z(\theta_2)^{1-\lambda} \end{aligned} $$ Since $Z(\theta_1), Z(\theta_2)\lt\infty$, therefore $Z(\theta)\lt\infty$, so $\theta\in\mathcal N$. The natural parameter space is a convex set.

Hölder's inequality was used in this proof, check https://mathworld.wolfram.com/HoeldersInequalities.html for more information

Take logarithm to above inequality, you will get log-partition function $A(\theta)$ a convex function.

$$ A(\theta) = A(\lambda\theta_1+(1-\lambda)\theta_2) \leq \lambda A(\theta_1) + (1-\lambda)A(\theta_2) $$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .