I am trying to prove something that I believe is related to the logsumexp soft maximum trick but I am getting stuck on formalising the bit where one can claim that one of the exponents dominates every other term in the summand. The question is as follows:

Suppose we have constants $\delta_i$ and $\alpha_{im}$, for $1 \leq i \leq n$, $1 \leq m \leq M$, with constraints

  1. $0 = \delta_1 < \delta_2 \leq \delta_3 \leq \dots \leq \delta_n$.
  2. $0 < \alpha_{im}$ for all $i, m$, and,
  3. $\alpha_{i1} + \alpha_{i2} + \dots + \alpha_{iM} = 1$ for all $i$.

Show that there exists a constant $C$ such that:

$$C = \lim_{k \rightarrow \infty} \frac{1}{k}\log\sum_m\left(\sum_i\beta_{im}e^{C - \delta_i}\right)^k,$$ where $\beta_{im} = (\alpha_{im})^{\frac{1}{k}}$.

What I did was to use the generalised Pascal's triangle to expand out the inner exponent so that:

$$\begin{split} \text{RHS} &= \lim_{k \rightarrow \infty} \frac{1}{k}\log\left(\sum_i(\sum_m \alpha_{im})e^{k(C-\delta_i)} + \text{lower order terms})\right)\\ &= \lim_{k \rightarrow \infty} \frac{1}{k}\log\left(\sum_ie^{k(C-\delta_i)} + \text{lower order terms})\right) \hspace{1cm}\text{using constraint (3)} \end{split}$$

The bit that I am stuck with is showing that there is a value of $C$ so that $\sum_ie^{k(C-\delta_i)} $ dominates the 'lower order terms', since, after that, we can reduce the problem to the soft maximum as discussed by John D. Cook:

$$\max(x_1, x_2, ...) = \lim_{k \rightarrow \infty} \operatorname{softmax}(k, x_1, x_2, ...)$$


$$\operatorname{softmax}(k, x_1, x_2, ...) = \frac{1}{k}\log(\exp(kx_1) + \exp(kx_2) + \dots)$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.