New answers tagged

1

As noted in my comment, it suffices to show the inequality for $x\in[0,\frac{1}{2}]$. Note that the inequality becomes an equality at the endpoints, $0$ and $\frac{1}{2}$. The inequality is tighter around $\frac{1}{2}$ than around $0$. In our proof, we will distinguish two (overlapping) cases, $x$ near $0$ or $x$ near $\frac{1}{2}$. When $x$ is near ...


1

is there a formal measure of bitwise entropy that takes into account these factors? You're using the term entropy in reference to some given finite string, whereas entropy is a function defined on probability distributions. One way of reconciling this is to suppose that the relevant probability distributions are the actual ("empirical") distributions of ...


0

The long and painful way: "differentiating, and differentiating." Define $f,g\colon (0,1)\to \mathbb{R}$ by $f(x) = 1-4\left(x-\frac{1}{2}\right)^2$ and $g(x) = 1-\left(1-\frac{x}{1-x}\right)^2$. We will show $$ h(x) \geq f(x) \geq g(x), \qquad x\in(0,1). $$ Claim. $h(x) \geq f(x)$ for all $x\in(0,1)$. Proof. Both functions are $C^\infty$, and we have ...


1

For example, let $b = 2$. You have 3 events $x_1, x_2, x_3$, $$p(x_1) = 0.2\\ p(x_2) = 0.5 p(x_3) = 0.3$$ Then $$H = -\sum_i p(x_i) \log_2(p(x_i)) = - p(x_1) \log_2(p(x_1))- p(x_2) \log_2(p(x_2))- p(x_3) \log_2(p(x_3)) =\\ 0.46 + 0.50 +  0.52 = 1.48 $$ Perhaps, if you tell me in which context you found that formula, I could give you a more meaningful ...


2

I do not think that this is elegant enough. Considering $$ h(x) = x \log_2\frac{1}{x}+(1-x) \log_2\frac{1}{1-x}$$ $$g(x)=1- \left(1-\frac{x}{1-x}\right)^2$$ $$f(x)=h(x)-g(x)$$ Expanding $f(x)$ as a Taylor series built at $x=\frac 12$, the result is $$f(x)= \left(16-\frac{2}{\log (2)}\right)\left(x-\frac{1}{2}\right)^2+64 \left(x-\frac{1}{2}\right)^3+ ...


0

The probability distribution of $Y|X^n=x$ converges to the probability distribution $Y|X=x$ for increasing $n$. Nevertheless, and that is the problem in this case, entropy is not continuous in a converging sequence of probability measures (see, e.g., this paper).


0

If $g$ maps between two sets that have the same dimension, the inequality may be strict. The inequality appears in Papoulis' book on Probability, Random Variables, and Stochastic Processes - an exact value, albeit depending on the function $g$ in a complicated way, can be found in Corollary 1 in this paper: $$h(Y) = h(X)+\int f_x\log |J|dx - H(X|Y). $$ If ...


0

The most straightforward approach is to compare the calculated distribution to your empirical distribution via a chi-square goodness of fit test. Since its a markov chain, you could test each conditional distribution (e.g., $p_{11},p_{12},p_{13},p_{14}$) separately. You'd need to verify that you have enough data so that the expected number of transition ...


1

The first inequality means that in general conditioning may reduce the information. If you prefer, note that $$H(X)\ge H(X|Z)$$ and then condition on $Z$ on both sides. For the identity, similarly, since $$H(X|Y)+H(Y)=H(X,Y),$$ after conditioning on $Z$ on both sides you get the inequality above.


1

$$\mathsf H(Y\mid X) = - \sum_x p_X(x) \sum_y p_{Y\mid X}(y\mid x)\log p_{Y\mid X}(y\mid x)$$ or $$\mathsf H(Y\mid X) ~=~ \sum_{x,y} p_{X,Y}(x,y)\log \frac{p_{X}(x)}{p_{X,Y}(x,y)}$$ Hint: (the latter will be a sum of six terms.)


0

I guess your main concern is whether $$ \limsup_{n \to \infty} a_n \leq \limsup_{n \to \infty} b_n, $$ given $a_n \leq b_n$. This is true since $$ \limsup_{n \to \infty} = \lim_{n \to \infty}\sup\{a_m : m \geq n\} \leq\lim_{n \to \infty} \sup\{b_m : m \geq n\} = \limsup_{n \to\infty} b_n. $$


0

I would consider the binary variables $V_k$ that are 1 when the keyword $K_k$ is present in a text randomly chosen and 0 otherwise. The mutual information for 2 keywords $K_a$,$K_b$ is : $\sum_{i,j}{P(V_a=i,V_b=j)}log{\frac{P(V_a=i,V_b=j)}{P(V_a=i)P(V_b=j)}}$ You can estimate it with : $F(0,0).log(\frac{F(0,0)}{F(0,.)F(.,0)} + ...


0

To prove that the von Neumann entropy (defined as $S(\rho)=-\mathrm{Tr}({\rho \ln \rho})$ with $\rho$ being the density matrix) is invariant under unitary change of basis, one should first realize what $\ln(\rho)$ stands for. The logarithm of a Hermitian matrix $\rho$ is defined as $$ \ln(\rho) = V \ln( V^\dagger \rho V) V^\dagger,$$ where $V$ is the ...


1

No, it does not imply that $h(A,T)=\infty$. A simple example is a disk $D$ and a rotation $T$ around the center of the disk. Of course, $T|A$ has zero entropy, but if you consider the radial projection $S$ to a smaller disk $B$ centered at the same point (that is, all in $A\setminus B$ projects to the boundary of $B$, while $S|B$ is the identity), then all ...


1

You can find the proof on the paper!


0

$N_2$ is a subgroup. In that case, $\pi \circ \pi_2$ will have uniform distribution on each coset, while the probability of each coset is the same for both $\pi \circ \pi_2$ and $\pi \circ \pi_1$. Note that $\pi_1$ does not need to be uniform. Also, by translation invariance, the result holds equally well if $N_2$ is a coset. (The same idea can be applied ...


2

I'm just going to focus on proving that for $|A|=5$, the $|E|=1$ system is not isomorphic to $|E|=2$ system. To do this I will count the total number of period $5$ elements, ie points where $T^5(x) = x$, and show that in the $|E|=1$ case there are countably many, whereas in the $|E|=2$ case there are uncountably many. Observation 1: Given a period $5$ ...


0

The chain rule for derivatives looks like this for physicists: $${\partial{\Phi}\over\partial{x}} = {\partial{\Phi}\over\partial{p}} {\partial{p}\over\partial{x}}$$ This means that ${\partial{\Phi}\over\partial{x}} = 0$ is a necessary condition for ${\partial{\Phi}\over\partial{p}} = 0$. I assume that ${\partial{\Phi}\over\partial{x}} = 0$ is the ...


0

A set of $n$ equiprobable events with total probability $p$ contributes $$ \sum_ip_i\log p_i=\sum_i\frac pn\log\frac pn=p\log\frac pn=p\log p-p\log n $$ to the negative entropy. In your case, all three sets have total probability $p=\frac13$.


0

EDIT: This is only an answer to the first version of the asker's question. This looks like physicists' notation, where p and x are codependent variables. If you want to derive this by $p$, you need to formulate the entire term as being dependent on only $p$ instead of $x$. If $p$ is dependent on $x$, you need to rewrite the dependency to make $x$ dependent ...



Top 50 recent answers are included