# Optimization with non-differentiable function (entropy)

The function $p:\mathbb{R}\to[0, 1]$ is such that $\sum_{x \in \mathcal{X} \subset \mathbb{R}}p(x)=1$, where $\mathcal{X}$ is a countable set.

Let $$H(p) = -\sum_{x \in \mathcal{X}}p(x)\log[p(x)]$$ where we define $p(x)\log[p(x)] = 0$ wherever $p(x) = 0$.

I am interested in solving for $p$ in the problem $$\max H(p) \text{ subject to }\sum_{x \in \mathcal{X}}p(x)=1\text{.}$$

My thoughts:

I can't use calculus techniques. Notably, $p$ is not continuous, and is therefore not differentiable. If this weren't the case, I would use Lagrange multipliers.

I've honestly never run into an optimization problem before where I couldn't use derivatives, so I'm not even sure where to begin.

• It isn't discrete. It is an infinite dimensional problem, as the variable is from $[0,1]^{\mathcal X}$. – user251257 Dec 21 '17 at 0:14
• @user251257 Ah, thanks for that. You're completely correct. Regardless, I can't use calculus to solve for $p$. – Clarinetist Dec 21 '17 at 0:14

Take $p_n = ({1 \over n},\cdots,{1\over n},0,\cdots)$, then $H(p_n) = -\log {1 \over n}$ and $\sup H(p_n) = \infty$, so $H$ is unbounded.