Hot answers tagged

4

I do not think that this is elegant enough. Considering $$ h(x) = x \log_2\frac{1}{x}+(1-x) \log_2\frac{1}{1-x}$$ $$g(x)=1- \left(1-\frac{x}{1-x}\right)^2$$ $$f(x)=h(x)-g(x)$$ Expanding $f(x)$ as a Taylor series built at $x=\frac 12$, the result is $$f(x)= \left(16-\frac{2}{\log (2)}\right)\left(x-\frac{1}{2}\right)^2+64 \left(x-\frac{1}{2}\right)^3+ ...


3

HINT: Let $F(d)$ be the function $$F(d)=\int_{-d}^d f(x-t)\,dt \tag 1$$ Expand $F(d)$ as given in $(1)$ in a Taylor series around $d=0$ and find $$F(d)=2f(x)d+\frac13 f''(x)d^3+O(d^5)$$ SPOILER ALERT: Scroll over the highlighted region to reveal the full expansion


2

$$h(x) = x \log_2\frac{1}{x}+(1-x) \log_2\frac{1}{1-x}\ge1- \left(1-\frac{x}{1-x}\right)^2$$ If we let $x=\frac{1+y}{2}$ and push through the algebra, the claim is equivalent to: $$1-\frac{1}{2}\log_2(1-y^2)-\frac{y}{2}\log_2\left(\frac{1+y}{1-y}\right)\ge1-\frac{4y^2}{(1-y)^2}$$ which can be rearranged to: $$\frac{8y^2}{(1-y)^2}\ge \log_2(1-y^2) + ...


2

Hopefully, this is right! Note that from the weighted AM-GM inequality, We have that $$h(x)=\log_2{\frac{1}{x^x(1-x)^{1-x}}} \ge \log_2\frac{1}{x^2+(1-x)^2}$$ Thus we have to show $$\left(1-\frac{x}{1-x}\right)^2 \ge 1-\log_2\frac{1}{2x^2-2x+1}=\log_2{(4x^2-4x+2)}$$ Substitute $x=\frac{a+1}{a+2}$, and we have $$f(a)=a^2 -\log_2\left(\frac{a^2}{(a+2)^2}+1 ...


2

As noted in my comment, it suffices to show the inequality for $x\in[0,\frac{1}{2}]$. Note that the inequality becomes an equality at the endpoints, $0$ and $\frac{1}{2}$. The inequality is tighter around $\frac{1}{2}$ than around $0$. In our proof, we will distinguish two (overlapping) cases, $x$ near $0$ or $x$ near $\frac{1}{2}$. When $x$ is near ...


2

I think you are attacking the problem using the wrong conceptual frame. The probability of decoding error for a repetition code of length $n$ (odd) corresponds to the event of majority of bit errors, that is, a Binomial: $$\delta = \sum_{k=(n+1)/2}^{n} \binom{n}{k} (1-p)^{n-k} p^{k} \tag{1}$$ This gives $\delta$ as a function of $p$ and $n$; and in ...


1

$$\frac{1}{2d}\int_{-d}^df(x-t)dt=\frac{1}{2d}\int_{-d}^d[f(x)-f'(x)t+\frac{1}{2}f''(x)t^2+\cdots]dt$$ $$=\frac{1}{2d}f(x)\times2d-\frac{1}{2d}\int_{-d}^df'(x)tdt+\frac{1}{2d}\int_{-d}^d\frac{1}{2}f''(x)t^2dt+\cdots$$ $$=f(x)-\frac{1}{2d}f'(x)\int_{-d}^dtdt+\frac{f''(x)}{4d}2\times\frac{d^3}{3}+\cdots$$ $$=f(x)+\frac{f''(x)}{6}d^2+\cdots$$


1

Take the Taylor expansion of $f(x-t)$ about $x$ upto the term in $x^3$ and integrate term by term to get: $$ \int_{-d}^df(x-t)dt\approx\int_{-d}^d \left[f(x)+(-t)f'(x)+\frac{t^2}{2}f''(x)\right]\;dt=2df(x)+\frac{d^3}{3}f''(x) $$ and the error term is $O(t^5)$


1

It is easier to separate the process using the chain rule instead of using the full equation for I: $$I=H(ap)-H(1-a)p$$ $$\frac{dI}{dp}=\frac{dH(ap)}{d(ap)}\frac{d(ap)}{dp}-H(1-a)=\frac{dH(ap)}{d(ap)}a-H(1-a)$$ Now we need to find $H'$ $$\frac{dH}{dx}=-log_2(x)-x\frac{1}{x ...


1

In the multi-parameter case, $\theta = (\theta_1, ..., \theta_k)$, Fisher information is a $k\times k$ matrix where the $ij$ entry is given by $$ \mathcal{I}(\theta)_{ij} = -E[\frac{\partial}{\partial\theta_i \partial \theta_j} \log f(X;\theta)], $$ so, assuming that the sample is i.i.d, the additivity is still holds component-wise in the information ...


1

Apologies in advance for not being able to format this properly. The definition of D is the expectation value, E, of the distortion measure. E[d(x, x^)] = p(1|0) * 1 = p(1|0)= D. Now, because X is Bernoulli with probability 1/2, we have that p(0) = 1/2 = p(0|0) + p(0|1) = p(0|0). Finally p(1) = 1/2 = p(1|0) + p(1|1) = D + p(1|1), so p(1|1) = 1/2 - D. ...


1

What is the name of this approach in the literature? This is called block entropy; however, it has two different interpretations, corresponding to the following two different scenarios. In both scenarios we have a fixed string of symbols $y_1..y_n$, each from a finite alphabet $A$, and $H(X)=-\ E\ \log(p(X))$ denotes the Shannon entropy of a random ...


1

The most general definition of information quantities is in terms of a KL divergence. This definition is discussed starting on page 107 of Entropy and Information Theory by Gray. It's a kind of pedantic definition. On a probability space $(\Omega,\mathcal{A})$ with probability measures $P,Q$ and a finite-alphabet RV $Z$, define the relative entropy of a ...



Only top voted, non community-wiki answers of a minimum length are eligible