Tag Info

1

As noted in my comment, it suffices to show the inequality for $x\in[0,\frac{1}{2}]$. Note that the inequality becomes an equality at the endpoints, $0$ and $\frac{1}{2}$. The inequality is tighter around $\frac{1}{2}$ than around $0$. In our proof, we will distinguish two (overlapping) cases, $x$ near $0$ or $x$ near $\frac{1}{2}$. When $x$ is near ...

1

is there a formal measure of bitwise entropy that takes into account these factors? You're using the term entropy in reference to some given finite string, whereas entropy is a function defined on probability distributions. One way of reconciling this is to suppose that the relevant probability distributions are the actual ("empirical") distributions of ...

0

The long and painful way: "differentiating, and differentiating." Define $f,g\colon (0,1)\to \mathbb{R}$ by $f(x) = 1-4\left(x-\frac{1}{2}\right)^2$ and $g(x) = 1-\left(1-\frac{x}{1-x}\right)^2$. We will show $$h(x) \geq f(x) \geq g(x), \qquad x\in(0,1).$$ Claim. $h(x) \geq f(x)$ for all $x\in(0,1)$. Proof. Both functions are $C^\infty$, and we have ...

1

For example, let $b = 2$. You have 3 events $x_1, x_2, x_3$, $$p(x_1) = 0.2\\ p(x_2) = 0.5 p(x_3) = 0.3$$ Then $$H = -\sum_i p(x_i) \log_2(p(x_i)) = - p(x_1) \log_2(p(x_1))- p(x_2) \log_2(p(x_2))- p(x_3) \log_2(p(x_3)) =\\ 0.46 + 0.50 + 0.52 = 1.48$$ Perhaps, if you tell me in which context you found that formula, I could give you a more meaningful ...

2

I do not think that this is elegant enough. Considering $$h(x) = x \log_2\frac{1}{x}+(1-x) \log_2\frac{1}{1-x}$$ $$g(x)=1- \left(1-\frac{x}{1-x}\right)^2$$ $$f(x)=h(x)-g(x)$$ Expanding $f(x)$ as a Taylor series built at $x=\frac 12$, the result is $$f(x)= \left(16-\frac{2}{\log (2)}\right)\left(x-\frac{1}{2}\right)^2+64 \left(x-\frac{1}{2}\right)^3+ ... 0 The probability distribution of Y|X^n=x converges to the probability distribution Y|X=x for increasing n. Nevertheless, and that is the problem in this case, entropy is not continuous in a converging sequence of probability measures (see, e.g., this paper). 0 If g maps between two sets that have the same dimension, the inequality may be strict. The inequality appears in Papoulis' book on Probability, Random Variables, and Stochastic Processes - an exact value, albeit depending on the function g in a complicated way, can be found in Corollary 1 in this paper:$$h(Y) = h(X)+\int f_x\log |J|dx - H(X|Y). $$If ... 0 The most straightforward approach is to compare the calculated distribution to your empirical distribution via a chi-square goodness of fit test. Since its a markov chain, you could test each conditional distribution (e.g., p_{11},p_{12},p_{13},p_{14}) separately. You'd need to verify that you have enough data so that the expected number of transition ... 1 The first inequality means that in general conditioning may reduce the information. If you prefer, note that$$H(X)\ge H(X|Z)$$and then condition on Z on both sides. For the identity, similarly, since$$H(X|Y)+H(Y)=H(X,Y),$$after conditioning on Z on both sides you get the inequality above. 1$$\mathsf H(Y\mid X) = - \sum_x p_X(x) \sum_y p_{Y\mid X}(y\mid x)\log p_{Y\mid X}(y\mid x)$$or$$\mathsf H(Y\mid X) ~=~ \sum_{x,y} p_{X,Y}(x,y)\log \frac{p_{X}(x)}{p_{X,Y}(x,y)}$$Hint: (the latter will be a sum of six terms.) 0 I guess your main concern is whether$$ \limsup_{n \to \infty} a_n \leq \limsup_{n \to \infty} b_n, $$given a_n \leq b_n. This is true since$$ \limsup_{n \to \infty} = \lim_{n \to \infty}\sup\{a_m : m \geq n\} \leq\lim_{n \to \infty} \sup\{b_m : m \geq n\} = \limsup_{n \to\infty} b_n. $$0 I would consider the binary variables V_k that are 1 when the keyword K_k is present in a text randomly chosen and 0 otherwise. The mutual information for 2 keywords K_a,K_b is : \sum_{i,j}{P(V_a=i,V_b=j)}log{\frac{P(V_a=i,V_b=j)}{P(V_a=i)P(V_b=j)}} You can estimate it with : F(0,0).log(\frac{F(0,0)}{F(0,.)F(.,0)} + ... 0 To prove that the von Neumann entropy (defined as S(\rho)=-\mathrm{Tr}({\rho \ln \rho}) with \rho being the density matrix) is invariant under unitary change of basis, one should first realize what \ln(\rho) stands for. The logarithm of a Hermitian matrix \rho is defined as$$ \ln(\rho) = V \ln( V^\dagger \rho V) V^\dagger,$$where V is the ... 1 No, it does not imply that h(A,T)=\infty. A simple example is a disk D and a rotation T around the center of the disk. Of course, T|A has zero entropy, but if you consider the radial projection S to a smaller disk B centered at the same point (that is, all in A\setminus B projects to the boundary of B, while S|B is the identity), then all ... 1 You can find the proof on the paper! 0 N_2 is a subgroup. In that case, \pi \circ \pi_2 will have uniform distribution on each coset, while the probability of each coset is the same for both \pi \circ \pi_2 and \pi \circ \pi_1. Note that \pi_1 does not need to be uniform. Also, by translation invariance, the result holds equally well if N_2 is a coset. (The same idea can be applied ... 2 I'm just going to focus on proving that for |A|=5, the |E|=1 system is not isomorphic to |E|=2 system. To do this I will count the total number of period 5 elements, ie points where T^5(x) = x, and show that in the |E|=1 case there are countably many, whereas in the |E|=2 case there are uncountably many. Observation 1: Given a period 5 ... 0 The chain rule for derivatives looks like this for physicists:$${\partial{\Phi}\over\partial{x}} = {\partial{\Phi}\over\partial{p}} {\partial{p}\over\partial{x}}$$This means that {\partial{\Phi}\over\partial{x}} = 0 is a necessary condition for {\partial{\Phi}\over\partial{p}} = 0. I assume that {\partial{\Phi}\over\partial{x}} = 0 is the ... 0 A set of n equiprobable events with total probability p contributes$$ \sum_ip_i\log p_i=\sum_i\frac pn\log\frac pn=p\log\frac pn=p\log p-p\log n  to the negative entropy. In your case, all three sets have total probability $p=\frac13$.

0

EDIT: This is only an answer to the first version of the asker's question. This looks like physicists' notation, where p and x are codependent variables. If you want to derive this by $p$, you need to formulate the entire term as being dependent on only $p$ instead of $x$. If $p$ is dependent on $x$, you need to rewrite the dependency to make $x$ dependent ...

Top 50 recent answers are included