# Questions tagged [information-theory]

The science of compressing and communicating information. It is a branch of applied mathematics and electrical engineering. Though originally the focus was on digital communications and computing, it now finds wide use in biology, physics and other sciences.

1,417 questions
0answers
12 views

### Difference Entropy Output and Input [on hold]

In a ternary channel with same probabilities for all inputs. I have H(X) entrance entropy and H(Y) output entropy. What does the difference of these 2 Entropy tell me?
0answers
17 views

### Majority voting 9 Bits Error probability

i have 1 Bit that will send 9 times. Each bit has an error probabiltiy of 1/3. The received Bits will be interpreted with a majority voting. Do anyone have an idea how i do check the probability ...
1answer
43 views

### Expected Hamming distance between typical strings

Let $S$ be the set of bitstrings of length $n$ with $pn$ ones and $(1-p)n$ zeros, for some $p \in [0,1]$. I want to show that, for large $n$, if one uniformly samples pairs from $(s,s') \in S^2$, ...
0answers
24 views

### Tricky information inequality $I(X;Z) \geq H(T)$

I am wondering whether $I(X; Z) \geq H(T)$ when the following conditions hold: $H(T | X) = H(T)$ $H(T | Y) = H(T)$ $H(T | X, Y) = 0$ $H(Y | Z) = H(T | Z) = 0$ $X, Y, Z, T$ are discrete. I know first ...
0answers
17 views

### Conditional Entropy sum of 2 dice rolls and even/uneven

So i have the sum of the 2 dice rolls with X and that the sum is even/uneven with Y out of 0 or 1 Probabilities: (2=1/36) (3=2/36) (4=3/36) (5=4/36) (6=5/36) (7=6/36) (8=5/36) (9=4/36) (1=3/36) (11=2/...
0answers
22 views

### understanding the conditional entropy in the case of having uniform distribution?

Would you please help me to understand the conditional entropy in this example which I got stuck in? The example Considers 4 uniformly popular binary vectors, for example; {f1,f2,f3,f4} each with ...
0answers
26 views

### What is your opinion on David Ellerman's Partition Logic and his distinction based view of Entropy? [on hold]

I have been skimming through David Ellerman's work on Partition Logic and Information theory. What exactly is the difference between dits and bits in his treatment of information theory? What exactly ...
1answer
28 views

### Probability conditional on entry of exchangeable random vector: does the index matter?

I have a random varbiable $Y\in\{0,1\}$, which is dependent on a random vector $X \in \{0,1\}^n$. Therebey, the entries in $X$ are exchangeable. That is, $P(X_i=1)=P(X_j=1)$. I want to compute/...
1answer
22 views

### Similarness of random samples

Say, I have a set of random variables, $\{\vec{x}_1,...,\vec{x}_N\}$, which were all drawn from the same probability distribution, $p(\vec{x})$. I do not know the form of the $p(\vec{x})$. I am given ...
0answers
17 views

### Probability mass of a source of L symbols

I am trying to get a list of probabilities to test a function that computes the mutual information of an information channel. I have this function in Matlab ...
0answers
32 views