1
$\begingroup$

Let $f:\mathcal{X}\rightarrow \mathcal{F}$ be a function with a unique, positive maximum at $x^*=\arg\sup_{x}f(x)$ where $\mathcal{X}$ and $\mathcal{F}$ are both bounded. Let $f$ be locally smooth about $x^*$, that is $\exists\ \Delta>0\ s.t. f(x)\in\mathbb{C}^2\ \forall\ x\in\{x\vert \lVert x-x^*\rVert<\Delta$ }. Is it possible to prove that the distribution $p_\epsilon(x)=z_\epsilon\exp\left(\frac{f(x)}{\epsilon}\right)$, where $z_\epsilon=\left(\int\exp\left(\frac{f(x)}{\epsilon}\right)dx\right)^{-1}$ is the normalisation constant, converges to a Dirac delta distribution in the limit $\epsilon\rightarrow 0$? I've been trying to formally show that

$$a)\quad\lim_{\epsilon\rightarrow0}\int z_\epsilon\exp\left(\frac{f(x)}{\epsilon}\right)\varphi(x)dx=\varphi(x^*)$$

for any continuous $\varphi(x)$, but I have limited background in analysis. Moreover, if this result holds, can it be extended to any joint $p_\epsilon(x)=z_\epsilon\exp\left(\frac{f(x)}{\epsilon}\right)q(x)$ with $z_\epsilon=\left(\int\exp\left(\frac{f(x)}{\epsilon}\right)q(x)dx\right)^{-1}$ where $q(x)$ is another distribution, that is can we prove:

$$b)\quad\lim_{\epsilon\rightarrow0}\int z_\epsilon\exp\left(\frac{f(x)}{\epsilon}\right)q(x)\varphi(x)dx=\varphi(x^*),$$

again for any $\varphi(x)$ is continuous?

For those interested, the context of this comes from reinforcement learning where we seek a policy $\pi(a\vert s)$ that is greedy with respect to a value function $Q(a,s)$, that is $\pi(a\vert s)=\delta(a\in\arg\max_{a'}Q(a',s))$.

Many thanks in advance.

$\endgroup$
  • $\begingroup$ That $f$ is bounded and continuous is not enough to make $\int \exp\left(\frac{f(x)}{\epsilon}\right) \, dx$ finite, so $p_\epsilon$ might not be defined. $\endgroup$ – md2perpe Apr 30 at 9:26
  • $\begingroup$ Ah, yeah forgot to say the domain of $f(x)$ is bounded too. Have edited. $\endgroup$ – Mattiegf Apr 30 at 9:48
  • $\begingroup$ I also think that $p_\epsilon \to \delta$ if $f$ has a maximum in one point. But I have not been able to show it yet. $\endgroup$ – md2perpe Apr 30 at 14:12
  • $\begingroup$ I've edited the question to make the notion of local smoothness more precise. $\endgroup$ – Mattiegf May 16 at 0:53
0
$\begingroup$

I think I've come up with a proof for a). Sorry if it is bit slow, computer scientists like things step-by-step!

Firstly, we define the auxiliary function to be
\begin{align} g(x):= f(x)-f(x^*). \end{align}

Note, $g(x)\le 0$ with equality at $g(x^*)=0$. Substituting $f(x)=g(x)+f(x^*)$ into $p_\varepsilon(x)$: \begin{align} p_\varepsilon(x)=&\frac{\exp\left(\frac{g(x)+f(x^*)}{\varepsilon}\right)}{\int_\mathcal{X}\exp\left(\frac{g(x)+f(x^*)}{\varepsilon}\right)dx},\\ =&\frac{\exp\left(\frac{g(x)}{\varepsilon}\right)\exp\left(\frac{f(x^*)}{\varepsilon}\right)}{\int_\mathcal{X}\exp\left(\frac{g(x)}{\varepsilon}\right)\exp\left(\frac{f(x^*)}{\varepsilon}\right)dx},\\ =&\frac{\exp\left(\frac{g(x)}{\varepsilon}\right)}{\int_\mathcal{X}\exp\left(\frac{g(x)}{\varepsilon}\right)dx}.\quad (1) \end{align} Now, substituting (1) into the limit in a) yields: \begin{align} \lim_{\varepsilon\rightarrow0}\int_{\mathcal{X}} \varphi(x)p_\varepsilon(x)dx=\lim_{\varepsilon\rightarrow0}\left(\int_{\mathcal{X}} \varphi(x)\frac{\exp\left(\frac{g(x)}{\varepsilon}\right)}{\int_\mathcal{X}\exp\left(\frac{g(x)}{\varepsilon}\right)dx}dx\right).\label{eq:limit_g} \end{align} Using the substitution $u:=\frac{(x^*-x)}{\sqrt{\varepsilon}}$ to transform the integrals in (1), we obtain \begin{align} \lim_{\varepsilon\rightarrow0}\int_{\mathcal{X}} \varphi(x)p_\varepsilon(x)dx&=\lim_{\varepsilon\rightarrow0}\left(\int_{\mathcal{U}} \varphi(x^*-\sqrt{\varepsilon}u)\frac{\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)}{\int_\mathcal{U}\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)\sqrt{\varepsilon}du}\sqrt{\varepsilon}du\right),\\ &=\lim_{\varepsilon\rightarrow0}\left(\frac{\int_{\mathcal{U}} \varphi(x^*-\sqrt{\varepsilon}u)\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)du}{\int_\mathcal{U}\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)du}\right).\quad (2) \end{align} We now find $\lim_{\varepsilon\rightarrow0}\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)$: Using $\mathrm{L'H\hat{o}pital's}$ rule to the second derivative with respect to $\sqrt{\epsilon}$, we have: \begin{align} \lim_{\varepsilon\rightarrow0}\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)&=\lim_{\varepsilon\rightarrow0}\left(\frac{\partial_{\sqrt{\varepsilon}}g(x^*-\sqrt{\varepsilon}u)}{\partial_{\sqrt{\varepsilon}}\varepsilon}\right),\\ &=\lim_{\varepsilon\rightarrow0}\left(\frac{\partial_{\sqrt{\varepsilon}}f(x^*-\sqrt{\varepsilon}u)}{\partial_{\sqrt{\varepsilon}}\varepsilon}\right),\\ &=\lim_{\varepsilon\rightarrow0}\left(\frac{-u^\top\nabla f(x^*-\sqrt{\varepsilon}u)}{2\sqrt{\varepsilon}}\right),\\ &=\lim_{\varepsilon\rightarrow0}\left(\frac{\partial_{\sqrt{\varepsilon}}\left(u^\top\nabla f(x^*-\sqrt{\varepsilon}u)\right)}{\partial_{\sqrt{\varepsilon}}(2\sqrt{\varepsilon})}\right),\\ &=\lim_{\varepsilon\rightarrow0}\left(\frac{u^\top\nabla^2 f(x^*-\sqrt{\varepsilon}u)u}{2}\right),\\ &=\frac{u^\top\nabla^2 f(x^*)u}{2}. \end{align} The integrand in the numerator of (2) therefore converges pointwise to $\varphi(x^*)\exp\left(\frac{u^\top\nabla^2 f(x^*)u}{2}\right)$, that is \begin{align} \lim_{\varepsilon\rightarrow0}\left(\varphi(x^*-\sqrt{\varepsilon}u)\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)\right)=\varphi(x^*)\exp\left(\frac{u^\top\nabla^2 f(x^*)u}{2}\right),\quad (3) \end{align} and the integrand in the denominator converges pointwise to $\exp\left(\frac{u^\top\nabla^2 f(x^*)u}{2}\right)$, that is \begin{align} \lim_{\varepsilon\rightarrow0}\left(\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)\right)=\exp\left(\frac{u^\top\nabla^2 f(x^*)u}{2}\right).\quad (4) \end{align} From the second order sufficient conditions for $f(x^*)$ to be a maximum, we have $u^\top\nabla^2 f(x^*)u \le 0$ $\forall\ u\in\mathcal{U}$ with equality only when $u=0$. This implies that (3) and (4) are both bounded functions.

By definition, we have $g(x^*-\sqrt{\epsilon} u)\le0\ \forall\ u\in\mathcal{U}$, which implies that $|\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)|\le1$. Consequently, the integrand in the numerator of (2) is dominated by $\lVert \varphi(\cdot)\rVert_\infty$, that is \begin{align} \left\lvert\varphi(x^*-\sqrt{\varepsilon}u)\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)\right\rvert\le\lVert \varphi(\cdot)\rVert_\infty,\quad (5) \end{align} and the integrand in the denominator is dominated by $1$, that is \begin{align} \left\lvert\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)\right\rvert\le1.\quad (6) \end{align}

Together (3)-(6) are the sufficient conditions for applying the dominated convergence theorem, allowing us to commute all limits and integrals in (2), yielding our desired result:

\begin{align} \lim_{\varepsilon\rightarrow0}\int_{\mathcal{X}} \varphi(x)p_\varepsilon(x)dx&=\lim_{\varepsilon\rightarrow0}\left(\frac{\int_{\mathcal{U}} \varphi(x^*-\sqrt{\varepsilon}u)\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)du}{\int_\mathcal{U}\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)du}\right),\\ &=\frac{\int_{\mathcal{U}}\lim_{\varepsilon\rightarrow0}\left( \varphi(x^*-\sqrt{\varepsilon}u)\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)\right)du}{\int_\mathcal{U}\lim_{\varepsilon\rightarrow0}\left(\exp\left(\frac{g(x^*-\sqrt{\varepsilon}u)}{\varepsilon}\right)\right)du},\\ &=\frac{\int_{\mathcal{U}} \varphi(x^*)\exp\left(u^\top\nabla^2 f(x^*)u\right)du}{\int_\mathcal{U}\exp\left(u^\top\nabla^2 f(x^*)u\right)du},\\ &=\varphi(x^*)\frac{\int_{\mathcal{U}} \exp\left(u^\top\nabla^2 f(x^*)u\right)du}{\int_\mathcal{U}\exp\left(u^\top\nabla^2 f(x^*)u\right)du},\\ &=\varphi(x^*). \end{align}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.