1
vote
1answer
53 views

Variational Methods, why KL divergence is the difference between true distribution and approximating distribution.

Likelihood = $L(\textbf{w}) = P(V\mid \textbf{w})$. $$\ln P(V\mid \textbf{w}) = \ln \sum_H P(H,V\mid \textbf{w})$$ $$= \ln \sum_H Q(H\mid V)\frac{P(H,V\mid \textbf{w})}{Q(H\mid V)}$$ $$\geq ...
2
votes
0answers
32 views

Joint entropy maximization with a constraint

So I have 2 random variables X and Y, where X can take on values {0,1,2,3} and Y can take on values {0,1,2,3,4}. I need to maximize H(X,Y) subject to the constraint that P(X≠Y)=0.5. This also gives ...
1
vote
0answers
46 views

Optimization problem in the Von Neumann Entropy

I have a constrainted optimization problem in the Von Neumann Entropy. In a CVX-like syntax the problem goes as follows: given variable $\mathtt{c(n)}$ $$\begin{align} \text{minimize} \qquad & ...
0
votes
1answer
119 views

Solving exponential equations for maxent with opensource applications or coding

I am working on maximum entropy. And I have a problem solving some equations. Look at these 2 equations: $\frac{(2*e^{-a*2-b*2^2}) +(6*e^{-a*6-b*6^2}) +(10*e^{-a*10-b*10^2}) ...
1
vote
0answers
282 views

Is conditional entropy a convex function?

A conditional entropy can be expressed in the following way, $H_{V_t}(V_s) = -\sum_{s,t}p(s,t)\log{p_t(s)} = -\sum_{s,t}p(s,t)\log{\frac{p(s,t)}{\sum_{s'}{p(s',t)}}}$ $s$ and $t$ are defined ...
3
votes
1answer
313 views

Does a maximum entropy probability distribution with KL-divergence constraint not exist?

In my earlier question I asked about a technical aspect of solving a system of equations arising from looking for an entropy-maximizing distribution $p(x)$ continuous on $\mathbb{R}$ and constrained ...