Maximum Entropy / Principle of Insufficient Reason I have a question about the maximum entropy principle and its implications.  Here's the problem that motivates my question:
Suppose, for instance, I have a random variable $X$ which takes on the value +1 with probability p and -1 with probability 1-p.  Knowing nothing else about the distribution, we might decide to maximize $H(p)=-\sum p_i \log p_i = -(p \log(p) + (1-p) \log (1-p))$ to determine p, which is pretty easy to do with basic calculus- we wind up with $p^*=\frac12$, which is the same result dictated by the Principle of Insufficient Reason.  It seems intuitive to me (this is where I think things start to get somewhat un-rigorous), that another way to think about this is to imagine that $p$ is itself a random variable drawn from $P  \approx \text{Uniform(0,1)}$, and so $p^*$ is simply the "most likely" value of $P$, $\mathbb{E}[P]=\frac12$, which agrees with our prior result.  
Now, here's my actual question.  Suppose I have some solid theoretical reason to assume that $X$ is more likely to assume +1 than -1, i.e. $p>1-p$.  When I try to maximize $H(P)$ under the constraint that $p>1/2$, the problem becomes ill-posed- the function approaches its maximum at the boundary condition, which it can never reach- the "answer" is $p^* = \frac12 + \epsilon$.  But, under the other interpretation holding p as a random variable, this problem can be posed as $p^* = \mathbb{E}[\max(P,1-P)]=\frac34$, which is a well-defined answer and seems like a "reasonable" solution: if we know that $p^*$ is between 1/2 and 1, then the midpoint of the two can't be an unreasonable guess.
Why does this seeming paradox occur?  Is there any theoretical answer that allows us to assign a "nice" probability like $\frac23$ or $\frac34$ to the question, or are we stuck with stating that the BC $p>1-p$ adds no new information, since the optimal entropy is still essentially the same?
 A: No, unfortunately, there isn't - this is a good example to showcase the problems one might face when trying to implement the maximum entropy principle, under inequality constraints (the initial formulation of the principle used only equality constraints). Still "the maximum entropy solution" is not that $p^*=1/2$, because our prior admitted information is that $p$ should be strictly higher than $1/2$.
Then, the solution is of the form $p^* = \frac 12 + \epsilon, \;\;\epsilon >0$, however inconvenient that might be. In practice, $\epsilon$ should be chosen as a very small number (but not zero), since by increasing its value we decrease the entropy of the distribution, in an unjustifiable way, given that we only know that $p> 1/2$, nothing more.  
But if $\epsilon$ is such a small number, does it really matter? It may matter a lot. The expected value of the random variable will be
$$E(X) = \left(\frac 12 + \epsilon\right)\cdot 1 + \left(1-\frac 12 - \epsilon\right)\cdot(-1) = 2\epsilon$$
Already we have doubled this very small number. Now assume that $X$ is the typical element of an i.i.d. stochastic process, which models the net emissions of a dangerous air-born chemical compound (in whatever suitable units), which is the by-product of one monthly cycle of an industrial process. The emissions are net, because the industry has in place a procedure to clean the air also (hence the possible value $-1$). We are interested in expected 10-year concentration, so we are interested in the sum of these ranodm variables, $Z = \sum_{i=1}^{120}X_i$. Under $p^*=1/2$, obviously $E(Z\mid p=1/2) = 0$. But under $p^* = \frac 12 + \epsilon$, we obtain
$$E(Z\mid p=1/2 + \epsilon) = 240\epsilon$$
This may be high, given any regulatory standards in place. Of course, the arbitrary value of $\epsilon$ makes things problematic: $0.001$ is a small number, but so is $0.00001$. What to do? Well, in practice again, one will attempt to acquire more information in order to bound $p$ from below with a sharp boundary value higher than $1/2$, by, for example, measuring the actual compound concentration for some period of time before concluding on the parameter value to be used.
