# Tagged Questions

The science of compressing and communicating information. It is a branch of applied mathematics and electrical engineering. Though originally the focus was on digital communications and computing, it now finds wide use in biology, physics and other sciences.

21 views

### Uniquely Decodable and Instantaneous

Which of the following codes are (a) uniquely decodable? (b) instantaneous? $C_1={00,01,0}$ $C_2={00,01,100,101,11}$ $C_3={0,10,110,1110,...}$ $C_4={0,00,000,0000}$ For part a, I think only $C_3$ ...
15 views

### Subfair odds and Kuhn-Tucker conditions

I'm reading Elements of Information Theory by Cover and Thomas and they touch on the fact that when computing the optimal betting strategy when the odds are subfair and one may not bet a certain ...
41 views

### Upper bound on Huffman codeword length

I am reading Elements of Information Theory by Cover and Thomas and have been unable to find an upper bound on the length of a codeword in a Huffman code, either in this book or on the web. Does one ...
30 views

### Optimal Betting on One Horse

A specific horse has odds o and you consider the chance of this horse winning to be p. You are given the opportunity to bet a fraction b of your money on this horse, while the remaining fraction 1 − b ...
30 views

### Huffman Code Assuming Wrong Distribution

Assume the random variable X ∼ p(x). We design a Huffman code C for this X but unfortunately we assume an incorrect probability distribution p' for the random variable. What can you say about the ...
37 views

24 views

### PRNG for compression

I'm trying to intuitively grasp information theory. You have a string of size X that contains a lot of information, say it's a movie. You have a string of size N << X which is going to be the ...
24 views

### Is relative entropy with respect to a pmf a continuous function?

Is the relative entropy $D(p || q)$ with a fixed pmf $q$, continuous over $p$, where $p \in \{x \in \mathbb{R}^n: \sum_{i=1}^n x_i = 1 , x_i \geq 0 \}$?
22 views

### Entropy of sum of uniform random variables on a simplex [duplicate]

For two i.i.d random variables $X$ and $Y$, which are uniformly distributed on the $n$-dimensional simplex $\Delta_n= \left\{(x_1,\ldots,x_n): x_i \geq 0, \sum_i x_i \leq 1 \right\}$, I want to find ...
49 views

53 views

### how to mathematically represent a matrix of vectors?

My problem is the following: I have a dataset in particular have $4$ dimensions, for didactic reasons I need to represent this dataset as a $m\times n$ matrix array such that the ($i$-th, $j$-th) ...
40 views

### Sets defined by probabilities

I am doing the following problem from Cover and Thomas, Elements of Information Theory, for self-study: Let $X_1,\dots, X_n$ be an i.i.d. sequence of discrete random variables with entropy $H(X)$. ...
69 views

64 views

### trouble understanding calculation of signal-to-noise for ldpc codes

My apologies if the answer to this question is too easy. I am a mathematics student and the subject of low density parity check codes is new to me. In many papers on LDPC codes, there are plots ...
75 views

### Capacity of AWGN channel with infinite bandwidth

I am supposed to find the capacity of an AWGN communication channel with infinite bandwidth $B$, signal power $S$ and spectral density of noise $n/2$. Now, I know that the formula for calculating ...
49 views

### Proving if a permutation cipher is perfectly secret?

From what I've read, perfect secrecy in its most basic form, that the encrypted text reveals no information about the plaintext, be it structure or content. A permutation cipher is easy for me to ...
73 views

### generalised log sum inequality

The log sum inequality states that $\sum_i a_i\ln\frac{a_i}{b_i}\geq a\ln{\frac{a}{b}}$ where $a=\sum_i a_i$ and $b=\sum_i b$. Is there a generalisation (with whatever conditions) that extends it ...
39 views

### calculus of variations or optimize over function form

I have a question about optimizing the following quantity over function form . Given unknown function $f(\theta)$ such that $f(\theta)\geqslant 0$ and $\int f(\theta)d\theta\leq \infty$. And ...
57 views

45 views

### higher moments of entropy… does the variance of $log x$ have any operational meaning?

The Shannon entropy is the average of the negative log of a list of probabilities $\{ x_1 , \dots , x_d\}$, i.e. $$H(x)= -\sum\limits_{i=1}^d x_i \log x_i$$ there are of course lots of nice ...
33 views

### Is there any definition of entropy of a stochastic process?

Entropy of finite random variables is defined in Wiki https://en.wikipedia.org/wiki/Entropy_(information_theory) Entropy rate of a stochastic process is defined in Wiki https://en.wikipedia.org/wiki/...
37 views

### Doubt in Conditional Probability

I'm studying Information theory from the book Information Theory, Coding and Cryptography-Rajan Bose. I got confused at one pos where they have derived the equation ...
39 views

31 views

### Comparison between Shannon's and Blackwell's measure of informativeness

I want to compare the concept of precision of information'' between signals $x \in X$ and states $\omega \in \Omega$ defined by Blackwell and Shannon. Denote the conditional probability ...