New answers tagged

0

You have already gotten some good answers, I thought I could add something more of use which is not really an answer, but maybe good if you find differential entropy to be a strange concept. Since we can not store a real or continuous number exactly, entropy for continuous distributions conceptually mean something different than entropy for discrete ...


2

Notice that $\ln(\color{blue}{\sqrt{\color{black}{x}}}) = \ln(x^{\color{blue}{\frac{1}{2}}}) = \color{blue}{\frac{1}{2}}\ln(x)$ for all $x > 0$. Using this identity, let us re-write the maximum entropy, $\frac{1}{2} + \ln(\sqrt{2\pi}\sigma)$, as follows: $$ \begin{align} \frac{1}{2} + \ln(\sqrt{2\pi}\sigma) &= \frac{1}{2} + ...


1

For continuous distribution like Normal/Gaussian we compute the differential entropy. You can find the derivation here http://www.biopsychology.org/norwich/isp/chap8.pdf For more info on differential entropy I recommend the book "Elements of Information Theory" by Cover and Thomas.


0

In $h(X) = -E[lnf(x)]$ the entropy is in nats, the units of entropy when you work with natural logarithm. In $h(X) = -E[logf(x)]$ the entropy is in bits, the units of entropy when you work with base 2 logarithm. $1 $nat = log$_2(e)$ bits, so...


1

There are many many positions that require 20 moves to solve, so I'd say the position that qualifies as "most scrambled" among them is the one that has fewest alternative paths to the sorted state. I have downloaded Herbert Kociemba's Cube Explorer (half-turn version) from http://kociemba.org/cube.htm and have been using it to track some 20-move positions. ...


1

Apologies in advance for not being able to format this properly. The definition of D is the expectation value, E, of the distortion measure. E[d(x, x^)] = p(1|0) * 1 = p(1|0)= D. Now, because X is Bernoulli with probability 1/2, we have that p(0) = 1/2 = p(0|0) + p(0|1) = p(0|0). Finally p(1) = 1/2 = p(1|0) + p(1|1) = D + p(1|1), so p(1|1) = 1/2 - D. ...


1

Getting rid of the syntactic sugar, we have to compute: $$ \int_{0}^{+\infty}x^{\alpha-1}e^{-\beta x}\log\left(x^{\alpha-1}e^{-\beta x}\right)\,dx = - \frac{\Gamma(\alpha+1)}{\beta^\alpha}+(\alpha-1)\int_{0}^{+\infty}x^{\alpha-1}e^{-\beta x}\log x\,dx$$ where: $$J(\alpha,\beta)=\int_{0}^{+\infty}x^{\alpha-1}e^{-\beta x}\log x\,dx = ...


1

The most general definition of information quantities is in terms of a KL divergence. This definition is discussed starting on page 107 of Entropy and Information Theory by Gray. It's a kind of pedantic definition. On a probability space $(\Omega,\mathcal{A})$ with probability measures $P,Q$ and a finite-alphabet RV $Z$, define the relative entropy of a ...


2

$$h(x) = x \log_2\frac{1}{x}+(1-x) \log_2\frac{1}{1-x}\ge1- \left(1-\frac{x}{1-x}\right)^2$$ If we let $x=\frac{1+y}{2}$ and push through the algebra, the claim is equivalent to: $$1-\frac{1}{2}\log_2(1-y^2)-\frac{y}{2}\log_2\left(\frac{1+y}{1-y}\right)\ge1-\frac{4y^2}{(1-y)^2}$$ which can be rearranged to: $$\frac{8y^2}{(1-y)^2}\ge \log_2(1-y^2) + ...


2

Hopefully, this is right! Note that from the weighted AM-GM inequality, We have that $$h(x)=\log_2{\frac{1}{x^x(1-x)^{1-x}}} \ge \log_2\frac{1}{x^2+(1-x)^2}$$ Thus we have to show $$\left(1-\frac{x}{1-x}\right)^2 \ge 1-\log_2\frac{1}{2x^2-2x+1}=\log_2{(4x^2-4x+2)}$$ Substitute $x=\frac{a+1}{a+2}$, and we have $$f(a)=a^2 -\log_2\left(\frac{a^2}{(a+2)^2}+1 ...


0

Let $X_1 = [X \ \ Y]$ and $X_2 = [X \ \ Z]$. Then $$H(f(X_1),g(X_1)) = H(f(X_1)) + H(g(X_1)|f(X_1))\\ H(f(X_1),g(X_2)) = H(f(X_1)) + H(g(X_2)|f(X_1))$$ so $$H(f(X_1),g(X_1))-H(f(X_1),g(X_2))= H(g(X_1)|f(X_1)) - H(g(X_2)|f(X_1)) \leq 0.$$ You can also view $f(X,Y),g(X,Z)$ as a version of $f(X,Y),g(X,Y)$ that has been passed through a channel with ...


-2

$$W=e^\frac{S}{k}$$ $W=e^\frac{S}{k}$


1

What is the name of this approach in the literature? This is called block entropy; however, it has two different interpretations, corresponding to the following two different scenarios. In both scenarios we have a fixed string of symbols $y_1..y_n$, each from a finite alphabet $A$, and $H(X)=-\ E\ \log(p(X))$ denotes the Shannon entropy of a random ...


2

As noted in my comment, it suffices to show the inequality for $x\in[0,\frac{1}{2}]$. Note that the inequality becomes an equality at the endpoints, $0$ and $\frac{1}{2}$. The inequality is tighter around $\frac{1}{2}$ than around $0$. In our proof, we will distinguish two (overlapping) cases, $x$ near $0$ or $x$ near $\frac{1}{2}$. When $x$ is near ...


2

is there a formal measure of bitwise entropy that takes into account these factors? You're using the term entropy in reference to some given finite string, whereas entropy is a function defined on probability distributions. One way of reconciling this is to suppose that the relevant probability distributions are the actual ("empirical") distributions of ...


0

The long and painful way: "differentiating, and differentiating." Define $f,g\colon (0,1)\to \mathbb{R}$ by $f(x) = 1-4\left(x-\frac{1}{2}\right)^2$ and $g(x) = 1-\left(1-\frac{x}{1-x}\right)^2$. We will show $$ h(x) \geq f(x) \geq g(x), \qquad x\in(0,1). $$ Claim. $h(x) \geq f(x)$ for all $x\in(0,1)$. Proof. Both functions are $C^\infty$, and we have ...


1

For example, let $b = 2$. You have 3 events $x_1, x_2, x_3$, $$p(x_1) = 0.2\\ p(x_2) = 0.5 p(x_3) = 0.3$$ Then $$H = -\sum_i p(x_i) \log_2(p(x_i)) = - p(x_1) \log_2(p(x_1))- p(x_2) \log_2(p(x_2))- p(x_3) \log_2(p(x_3)) =\\ 0.46 + 0.50 +  0.52 = 1.48 $$ Perhaps, if you tell me in which context you found that formula, I could give you a more meaningful ...



Top 50 recent answers are included