# Tagged Questions

The science of compressing and communicating information. It is a branch of applied mathematics and electrical engineering. Though originally the focus was on digital communications and computing, it now finds wide use in biology, physics and other sciences.

7 views

### resource for derivation showing the computing of mutual information for normal random variables

If I have 2 correlated normal random variables, and they are not be jointly normally distributed, is there a closed form answer for their mutual information? I've seen that if two normal random ...
4 views

### Convex conjugate of average Fisher information measure

What is a possible convex conjugate of the function $\rho \mapsto \int (\nabla \log \rho(x))^2 \rho(x) dx$? (Suppose $\rho$ is a sufficiently integrable probability density function on a $d$-...
27 views

29 views

### How to use the log-sum inequality to prove convexity of KL-divergence?

I'm trying to read up on information theory, and found the following: http://homes.cs.washington.edu/~anuprao/pubs/CSE533Autumn2010/lecture3.pdf Which states that the convexity of KL-divergence can ...
32 views

### Is there a name for the property of a code where symbol “space” is left unused?

For example, say I have the symbols A, B, C and D. If I encode these as A = 1, B = 01, C = 001 and D = 0001 (for a very simple example), I have a very simple prefix code. However, I know straight ...
41 views

21 views

### Deriving the Power Spectral Density of a Maximum Entropy Process

In Elements of Information Theory, Chapter 12, Section 6 Burg's Theorem is derived: Presented with the first $p$ values of the autocovariance function $R(k) = E[X_i X_{i+k}]$ a stochastic process ...
16 views

### How can I show that any integrable Passband or Baseband signal is also a finite energy signal?

I have supposed that, as the definition of a baseband/passband signal says, the function x(t) is integrable, continuous and bounded due to the fact that it forms a Fourier Transform pair with x'(f) (...
52 views

### information measure for matrix that is analogous to rank

Is there a measure for matrix that is analogous to rank of the matrix, but it is continuous on matrix elements? Say, we could say the information in identity matrix $I_n$ is $n$, and when the off-...
23 views

81 views

### Concavity of Shanon's information

It is known fact for random variables $(X,Y) \sim p(x,y)=p(x)p(y|x)$ the mutual information is concave function of $p(x)$ for fixed $p(y|x)$. I have two confusions in interpreting the above fact: 1) ...
47 views

### why can't sort 12 elements in 29 comparisons

The information theoretic lower bound for sorting 12 elements is using 29 comparisons, but actually we can't sort them in less than 30 comparisons. My problem is that why we can't reach the ...
33 views

### Does the following function define a distance metric?

For real numeric vectors of length $N$, let $a_n \succ b_n$ be one if true and zero if false. The distance between $A$ and $B$ is $$\sum_1^N a_n \succ b_n$$ Note that this is very similar to the ...
50 views

### Proof of Cyclic Redundancy Check validity

I'm looking to understand the use of a Cyclic Redundancy Check, in combination with the mathematics behind it. So far I have 1) For any message $$M(x)\cdot x^n = Q(x)G(x) + R(x)$$ Where $Q(x)$ is ...
14 views

### Specific examples of Side Information?

I'm starting to apply information theory to gambling. There is something called Side information (see details in [1]), which I understand is additional information about the outs of the game. It could ...
36 views

### Conceptual Question : Relationship between entropy and a technique for source coding

I want to encode the messages to a sequence of 1s and 0s (subsequently called "bits"). This is called "source coding". Shannon's source coding theory states that the entropy of a source that emits a ...
23 views

### Proving some inequalities related to Information Theory

I've been working on some inequalities related to the information theory section of my decision theory course, and I could use some help on some of the derivations for one of the inequalities. As a ...
21 views

### Kullback-Leibner divergence true distribution

I have an image with an object which I treat as 2-dimensional Gaussian random vector with mean equal to the center of the object surrounded by, roughly, 3-sigma ellipsoid. On the other hand I feed the ...
417 views

### Do Gödel numbers have a practical use?

Is there any example of Gödel numbers being actually used in practice? If so for what purpose?
23 views

### For convex $f$, why is $(p,q) \mapsto q \, f(p/q)$ convex on $\mathbb{R}_+^2$?

This fact was stated in the Wikipedia article on $f$-divergences to explain why they are jointly convex.
15 views

### Lower bound for limit of length of codeword

Here is the question I'm trying to solve. I don't really have any idea how to approach it/what theorem to use. For $p, \lambda >0$, let $m(n,p,\lambda)$ be defined to be the least $m$ ...
25 views

### Significance of Convex Sets for I-Projection

I have been reviewing the literature on information theoretic methods in statistics, and in particular, the method of I-projections. Given a discrete, finite alphabet $\mathcal{X}$, let $\prod$ denote ...
17 views

### How to prove $2d_H(\{XY\},\{X\}\{Y\})^2 \le I(X,Y)$?

Let $X$ and $Y$ be discrete random variables. Denote the joint distribution of $X$ and $Y$ by $\{XY\}$ and their marginal distributions by $\{X\}$ and $\{Y\}$. Let $\{X\}\{Y\}$ denote the product of ...
23 views

### Kullback-Leibler Divergence (KL) and Approximation Symmetry Property

The Kullback-Leibler Divergence doesn't satisfy the symmetric property. But, it can be approximated (bounded) to such a value. in this paper: Compressing Interactive Communication under product ...
11 views

### Checking if a code can be unambiguously decoded

The source of information is A = {a, b, c, d}. More info is given in the table below. I have to find the average length of the codes, compare it to the entropy of ...
71 views

### $λ=log(2)$ for the tent map – which basis for the logarithm?

If $\lambda$ is the largest positive Lyapunov exponent of a piecewise linear dynamical chaotic discrete in time map, then is there a relationship between the entropy $h$ and its $\lambda$. According ...
116 views

### Guess the number despite false answer

This is the Guess-The-Number game with a twist! Variant 1 Take any positive integer $n$. The game-master chooses an $n$-bit integer $x$. The player makes queries one by one, each of the ...