# Tagged Questions

The science of compressing and communicating information. It is a branch of applied mathematics and electrical engineering. Though originally the focus was on digital communications and computing, it now finds wide use in biology, physics and other sciences.

30 views
+50

29 views

### How to use the log-sum inequality to prove convexity of KL-divergence?

I'm trying to read up on information theory, and found the following: http://homes.cs.washington.edu/~anuprao/pubs/CSE533Autumn2010/lecture3.pdf Which states that the convexity of KL-divergence can ...
33 views

### Is there a name for the property of a code where symbol “space” is left unused?

For example, say I have the symbols A, B, C and D. If I encode these as A = 1, B = 01, C = 001 and D = 0001 (for a very simple example), I have a very simple prefix code. However, I know straight ...
41 views

16 views

### How can I show that any integrable Passband or Baseband signal is also a finite energy signal?

I have supposed that, as the definition of a baseband/passband signal says, the function x(t) is integrable, continuous and bounded due to the fact that it forms a Fourier Transform pair with x'(f) (...
23 views

82 views

### Concavity of Shanon's information

It is known fact for random variables $(X,Y) \sim p(x,y)=p(x)p(y|x)$ the mutual information is concave function of $p(x)$ for fixed $p(y|x)$. I have two confusions in interpreting the above fact: 1) ...
84 views

### Greater/lesser search with one false answer allowed

It is well known that you can determine the values of $n\geq 2$ bits using $k$ yes/no questions about the bits (for example, "is $x_1 \oplus x_3 = 1$?), even if one (but not more) of the answers ...
33 views

### Does the following function define a distance metric?

For real numeric vectors of length $N$, let $a_n \succ b_n$ be one if true and zero if false. The distance between $A$ and $B$ is $$\sum_1^N a_n \succ b_n$$ Note that this is very similar to the ...
50 views

### Proof of Cyclic Redundancy Check validity

I'm looking to understand the use of a Cyclic Redundancy Check, in combination with the mathematics behind it. So far I have 1) For any message $$M(x)\cdot x^n = Q(x)G(x) + R(x)$$ Where $Q(x)$ is ...
116 views

### Guess the number despite false answer

This is the Guess-The-Number game with a twist! Variant 1 Take any positive integer $n$. The game-master chooses an $n$-bit integer $x$. The player makes queries one by one, each of the ...
36 views

### Conceptual Question : Relationship between entropy and a technique for source coding

I want to encode the messages to a sequence of 1s and 0s (subsequently called "bits"). This is called "source coding". Shannon's source coding theory states that the entropy of a source that emits a ...
14 views

### Specific examples of Side Information?

I'm starting to apply information theory to gambling. There is something called Side information (see details in [1]), which I understand is additional information about the outs of the game. It could ...
23 views

### Proving some inequalities related to Information Theory

I've been working on some inequalities related to the information theory section of my decision theory course, and I could use some help on some of the derivations for one of the inequalities. As a ...
21 views

### Kullback-Leibner divergence true distribution

I have an image with an object which I treat as 2-dimensional Gaussian random vector with mean equal to the center of the object surrounded by, roughly, 3-sigma ellipsoid. On the other hand I feed the ...
mutual information of tow variables is $\displaystyle\sum\sum p(x,y)\ln\frac{p(x,y)}{p(x)p(y)}$ what is the mutual information of three variables? is it \$\displaystyle\sum\sum\sum p(x,y,z)\ln\frac{p(...