# Tag Info

4

I do not think that this is elegant enough. Considering $$h(x) = x \log_2\frac{1}{x}+(1-x) \log_2\frac{1}{1-x}$$ $$g(x)=1- \left(1-\frac{x}{1-x}\right)^2$$ $$f(x)=h(x)-g(x)$$ Expanding $f(x)$ as a Taylor series built at $x=\frac 12$, the result is $$f(x)= \left(16-\frac{2}{\log (2)}\right)\left(x-\frac{1}{2}\right)^2+64 \left(x-\frac{1}{2}\right)^3+ ... 3 HINT: Let F(d) be the function$$F(d)=\int_{-d}^d f(x-t)\,dt \tag 1$$Expand F(d) as given in (1) in a Taylor series around d=0 and find$$F(d)=2f(x)d+\frac13 f''(x)d^3+O(d^5)$$SPOILER ALERT: Scroll over the highlighted region to reveal the full expansion 2$$h(x) = x \log_2\frac{1}{x}+(1-x) \log_2\frac{1}{1-x}\ge1- \left(1-\frac{x}{1-x}\right)^2$$If we let x=\frac{1+y}{2} and push through the algebra, the claim is equivalent to:$$1-\frac{1}{2}\log_2(1-y^2)-\frac{y}{2}\log_2\left(\frac{1+y}{1-y}\right)\ge1-\frac{4y^2}{(1-y)^2}$$which can be rearranged to:$$\frac{8y^2}{(1-y)^2}\ge \log_2(1-y^2) + ...

2

Hopefully, this is right! Note that from the weighted AM-GM inequality, We have that $$h(x)=\log_2{\frac{1}{x^x(1-x)^{1-x}}} \ge \log_2\frac{1}{x^2+(1-x)^2}$$ Thus we have to show $$\left(1-\frac{x}{1-x}\right)^2 \ge 1-\log_2\frac{1}{2x^2-2x+1}=\log_2{(4x^2-4x+2)}$$ Substitute $x=\frac{a+1}{a+2}$, and we have $$f(a)=a^2 -\log_2\left(\frac{a^2}{(a+2)^2}+1 ... 2 As noted in my comment, it suffices to show the inequality for x\in[0,\frac{1}{2}]. Note that the inequality becomes an equality at the endpoints, 0 and \frac{1}{2}. The inequality is tighter around \frac{1}{2} than around 0. In our proof, we will distinguish two (overlapping) cases, x near 0 or x near \frac{1}{2}. When x is near ... 2 I think you are attacking the problem using the wrong conceptual frame. The probability of decoding error for a repetition code of length n (odd) corresponds to the event of majority of bit errors, that is, a Binomial:$$\delta = \sum_{k=(n+1)/2}^{n} \binom{n}{k} (1-p)^{n-k} p^{k} \tag{1}$$This gives \delta as a function of p and n; and in ... 1$$\frac{1}{2d}\int_{-d}^df(x-t)dt=\frac{1}{2d}\int_{-d}^d[f(x)-f'(x)t+\frac{1}{2}f''(x)t^2+\cdots]dt=\frac{1}{2d}f(x)\times2d-\frac{1}{2d}\int_{-d}^df'(x)tdt+\frac{1}{2d}\int_{-d}^d\frac{1}{2}f''(x)t^2dt+\cdots=f(x)-\frac{1}{2d}f'(x)\int_{-d}^dtdt+\frac{f''(x)}{4d}2\times\frac{d^3}{3}+\cdots=f(x)+\frac{f''(x)}{6}d^2+\cdots$$1 Take the Taylor expansion of f(x-t) about x upto the term in x^3 and integrate term by term to get:$$ \int_{-d}^df(x-t)dt\approx\int_{-d}^d \left[f(x)+(-t)f'(x)+\frac{t^2}{2}f''(x)\right]\;dt=2df(x)+\frac{d^3}{3}f''(x) $$and the error term is O(t^5) 1 It is easier to separate the process using the chain rule instead of using the full equation for I:$$I=H(ap)-H(1-a)p\frac{dI}{dp}=\frac{dH(ap)}{d(ap)}\frac{d(ap)}{dp}-H(1-a)=\frac{dH(ap)}{d(ap)}a-H(1-a)$$Now we need to find H'$$\frac{dH}{dx}=-log_2(x)-x\frac{1}{x ...

1

In the multi-parameter case, $\theta = (\theta_1, ..., \theta_k)$, Fisher information is a $k\times k$ matrix where the $ij$ entry is given by $$\mathcal{I}(\theta)_{ij} = -E[\frac{\partial}{\partial\theta_i \partial \theta_j} \log f(X;\theta)],$$ so, assuming that the sample is i.i.d, the additivity is still holds component-wise in the information ...

1

Apologies in advance for not being able to format this properly. The definition of D is the expectation value, E, of the distortion measure. E[d(x, x^)] = p(1|0) * 1 = p(1|0)= D. Now, because X is Bernoulli with probability 1/2, we have that p(0) = 1/2 = p(0|0) + p(0|1) = p(0|0). Finally p(1) = 1/2 = p(1|0) + p(1|1) = D + p(1|1), so p(1|1) = 1/2 - D. ...

1

What is the name of this approach in the literature? This is called block entropy; however, it has two different interpretations, corresponding to the following two different scenarios. In both scenarios we have a fixed string of symbols $y_1..y_n$, each from a finite alphabet $A$, and $H(X)=-\ E\ \log(p(X))$ denotes the Shannon entropy of a random ...

1

The most general definition of information quantities is in terms of a KL divergence. This definition is discussed starting on page 107 of Entropy and Information Theory by Gray. It's a kind of pedantic definition. On a probability space $(\Omega,\mathcal{A})$ with probability measures $P,Q$ and a finite-alphabet RV $Z$, define the relative entropy of a ...

Only top voted, non community-wiki answers of a minimum length are eligible