# Tagged Questions

The science of compressing and communicating information. It is a branch of applied mathematics and electrical engineering. Though originally the focus was on digital communications and computing, it now finds wide use in biology, physics and other sciences.

34 views

### Maximizing sum of logarithms (Z-channel capacity)

In the context of information theory, I am trying to maximize the following function (mutual information of the Z-channel's input and output) with respect to $p$ in order to derive Z-channel's ...
44 views

### Relative entropy (KL divergence) of sum of random variables

Suppose we have two independent random variables, $X$ and $Y$, with different probability distributions. What is the relative entropy between pdf of $X$ and $X+Y$, i.e. $$D(P_X||P_{X+Y})$$ assume all ...
32 views

22 views

17 views

### Channel capacity of sum of symmetric channels

I've got a channel matrix $P$ of the form $\begin{bmatrix} Q \\ R \end{bmatrix}$ where $Q,R$ are channel matrices of symmetric channels, so they now have different input alphabets but the ...
28 views

### Basic Entropy Inequality and Identity question

This is a solution to a problem I am working on: \begin{aligned} H(X|Y) + H(Y|Z) &\ge^? H(X|Y, Z) + H(Y|Z) \\ &=^\text{?}H(X,Y |Z) \\ &= H(X|Z) + H(Y|X, Z)\\ &\ge H(X|...
71 views

### Why do we like sticking random variables into their own distributions?

Let $X$ be a random variable taking values in the set $S$. It has some distribution $f(s)$. Often in statistics, we are interested in the real valued random variable $f(X)$. Here are some examples: ...
12 views

### Representation of the optimal filter measure as the measure of a diffusion process

In "Mitter SK, Newton NJ. A Variational Approach to Nonlinear Estimation. SIAM J Control Optim. 2003 Jan;42(5):1813–33", it is shown that the path estimation measure $P_{X|Y}(\cdot,y)$ for the ...
26 views

### Calculating Entropy and Information Gain of a Variable

I have the following values for two random variables. I need to compute the following values: a. H(Y) b. H(Y|X) c. and finally IG(Y|X) I will show what I have calculated so far. a. H(Y) = -(.5*...
12 views

### Removing the dimension factor in Fannes inequality

Given two distributions $x=(x_1,\ldots, x_n),y=(y_1,\ldots y_n)$ on $[n]$, it is known by Fannes inequality that $H(x)-H(y)\leq O(\|x-y\|_1\log n)$, where $H(\cdot)$ and $\|\cdot\|_1$ represent ...
11 views

### Closed form of Mutual Information, Continuous Random Variables

Is there any closed form for any non Gaussian Joint distribution ? For the Gaussian case $I(X,Y)=f( \varrho )$ where $\varrho$ is the correlation coefficient, and $f$ is an known increasing ...
31 views

56 views

### Does it pay to know what you know?

Let's play a game. I ask you question a yes/no question, and you answer. You don't answer with a yes or no though, you answer with a probability of it being yes ($P \in (0,1)$). For example, I might ...
28 views

An optional challenge assignment: Given a stationary Markov chain $\mathbf X=(X_k)^\infty_{k=1}$ where $X_k$ takes values in {0,1,2}. Let it have a probability transition matrix $P=[P_{ij}]=Pr(X_{k+1}=... 1answer 39 views ### What is the link between homomorphisms and mutual information? Intuitively, there seems to be a link between the (kind of) homomorphism between two algebraic structures and the mutual information between two variables. However, since I'm not a mathematician, it's ... 0answers 26 views ### Inference on a factor graph (Sum-product Algorithm) I was going through the sum-product algorithm which can be used to find marginal distribution efficiently(and exactly) when the factor graph is a tree. I found it difficult to understand the way they ... 0answers 24 views ### Cool property of KL divergence, help me fix my reasoning So for any rv$X$and any event$E$the following property should hold for KL divergence: $$\log \frac{1}{P_X(E)} = D(P_{X|X\in E} \| P_X)$$ I think this is pretty remarkable, but I don't seem to be ... 1answer 45 views ### Rate distortion function with infinite distortion I am working through the problems in Elements of Information Theory by Cover and Thomas and have come across the following problem I couldn't answer. The problem is to find the rate distortion ... 1answer 21 views ### Does the Information Gain algorithm favor a high-entropy attribute or a low-entropy one? This might not be mutual to mathematics but it does relate to Information-Theory. My question is: Does the InformationGain algorithm, in Decision-Tree machine-learning, favor a high-entropy ... 0answers 33 views ### Channels with memory have higher capacity I am working through Elements of Information Theory by Cover and Thomas and have come across the following solution to one of their problems that I don't understand. Consider a binary, symmetric ... 3answers 50 views ### “Self-referential” probability mass functions I am currently self-studying information theory from "Quantum Information Theory" by Mark M. Wilde. He uses a kind of notation that I don't understand at all. I will explain the problem using ... 1answer 98 views ### Mutual information vs Information Gain I always thought that mutual information and information gain refer to the same thing, however looking at Wikipedia: http://en.wikipedia.org/wiki/Information_gain https://en.wikipedia.org/wiki/... 0answers 62 views ### Generalization of Shannon's source coding theorem with a posteriori entropies This doubt is with reference to section 5-5 of "Information theory and Coding" by Prof. Norman Abramson. Under the topic "A generalization of Shannon's First Theorem", the text discusses how knowledge ... 3answers 84 views ### Greater/lesser search with one false answer allowed It is well known that you can determine the values of$n\geq 2$bits using$k$yes/no questions about the bits (for example, "is$x_1 \oplus x_3 = 1$?), even if one (but not more) of the answers ... 2answers 72 views ### Deducing an integer from$0$-$15$and lying I'm interested in reducing the upperbound of the number of questions needed and in finding alternate solutions to solve the following question: Suppose I have thought up an integer between$0$and ... 0answers 22 views ### Showing that for a family of subsets of$[n]$enough elements appear in high frequencies Let$\mathcal{F} \subseteq 2^{[n]}$a familiy of subsets. Assume that the following applies: For every$A \subseteq [n]$, such that$|A|\leq \alpha n$($\alpha > 0$is given), there's a subset$...
Prove that code C is uniquely decodable if the extension $C^k(x_1,x_2,...,x_k)=C(x_1)C(x_2)...C(x_k)$ is a one-to-one mapping from $\mathcal{X}^k$ to $D^*$ for every $k\geq1$. I know that for ...
Which of the following codes are (a) uniquely decodable? (b) instantaneous? $C_1={00,01,0}$ $C_2={00,01,100,101,11}$ $C_3={0,10,110,1110,...}$ $C_4={0,00,000,0000}$ For part a, I think only $C_3$ ...