The science of compressing and communicating information. It is a branch of applied mathematics and electrical engineering. Though originally the focus was on digital communications and computing, it now finds wide use in biology, physics and other sciences.

learn more… | top users | synonyms (1)

1
vote
0answers
38 views

A probabilty of error calculation

Let's assume I have $N$ binary strings $\{T_1,T_2,\ldots,T_N\}$ of length $L$. All these strings satisfy a minimum hamming distance with respect to a reference binary string R with $\|R\|_1$ ones and ...
0
votes
0answers
11 views

Fewest number of questions to find subinterval

I need some help with the following question: Let $S=${$x_1$,... $x_n$} be a set of $n$ real numbers listed in ascending order: $x_1<x_2<...<x_n$. Given a real number $r$ that exists in ...
4
votes
1answer
24 views

Applications of information theory in economics?

What are some direct applications of information theory in economics theory and/or finance? Any relevant articles, surveys, or book references are appreciated (especially if they are targeted to ...
0
votes
1answer
14 views

High mutual information = (almost) deterministic relationship?

If two random variables $X, Y$ have high mutual information $I(X;Y)$, intuitively does that mean $X,Y$ have almost deterministic relationships, say $Y=f(X)+\epsilon$ where $\epsilon$ is a noise random ...
0
votes
1answer
22 views

What is the sum-capacity for a non-symmetric interference channel for information theorists?

This question is dedicated for people who are experts in information theory. An interesting result for a two user interference channel in information theory, is the sum-capacity to within one bit. It ...
2
votes
3answers
53 views

Code is not cyclic for any q

I have code $C$ over $F_p$ with generator matrix which looks like $G = \begin{pmatrix} 0 &0& 0& 1& 0& 1& 1 &1\\ 1& 0 &0& 0 &1 &0 &1& 1\\ ...
0
votes
0answers
24 views

Information Content of an event and its probability

From a group of candidates, 25% are not suitable for the University admission. As the result of selection process only 75% of these unsuitable students are rejected. Overall 50% of students are ...
0
votes
1answer
32 views

Entropy of noisy signal

We have input signal $X$, the output signal Y and random noise $Z$, then: $$Y=X+Z$$ Of course, the mutual entropy: $$I(Y,X)=H(X)-H(X\mid Y)=H(X)-H(X-Y\mid Y) \geq H(X)-H(X-Y)$$ Could we say that ...
0
votes
1answer
45 views

Conditional Entropy in rolling a dice

A 6-sided die is tossed once. Two events X and Y are defined. X is the event in which an even number comes up and Y is the event in which the number is a multiple of 3. The value of H(X|Y) needs to be ...
2
votes
0answers
17 views

How to deconstruct Shannon Joint Entropy H(X,Y,Z) equation for semi-related variables?

Background The purpose of this is to produce a shuffle correction for Transfer Entropy estimation: TEx->y = H(Xt+τ) - H(Xt) - H(Xt+τ,Yt,Xt) + H(Xt,Yt) In order to produce a shuffling ...
1
vote
1answer
25 views

What does it mean for something to be “more” Gaussian?

I'm currently reading this paper on information theory and the brain. Within the text (p. 16) they say: It is important to note that the above expression [the mutual information between a signal ...
1
vote
1answer
13 views

If gen matrix has even weigth rows, do codewords have even weigth for non binary code?

Is that true that in a non binary code C every codeword has even weight if and only if every row of G has even weight?
1
vote
1answer
17 views

Subset of linear dual code

Hi I need to show that $$C_1 \subseteq C_2 \Leftrightarrow C_2^{⊥} \subseteq C_1^{\perp}$$ In guess I need to use standard form matrices for generator matrix and parity check matrix(also parity ...
0
votes
2answers
21 views

(7,4) Hamming Code with 2 bit errors

I need help proving that a HammingCode with 2 bits flipped can create a new codeword by flipping just one more bit. I worked through the problem and created an example of my own but am having ...
1
vote
1answer
25 views

Channel Capacity of a Cycle Graph

I have the following problem: Given a discrete memoryless channel $Y = X + Z \mod5$, where $X$ is selected from one of 5 symbols (0, 1, 2, 3, 4), $Z$ randomly selected from (-1, 0, 1), and $X$ and $Z$ ...
0
votes
1answer
16 views

prove a theorem about an upper bound of entropy of a random vector

There is a theorem that: if Z is any zero-mean, complex random vector with covariance $E[ZZ^H]=R_z$, then $H(Z)\leq \log|{\pi eR_z}|$, with equality holding if and only if Z has a circularly ...
0
votes
0answers
16 views

Analytical calculation of mutual information of a sine time series

Suppose I have the following time series $$ x_t=\sin(0.02\pi t) $$ and I want to calculate the mutual information $I_{x_t,x_{t-\tau}}$ which by definition is $$ ...
0
votes
0answers
13 views

estimating mutual information via nearest neighbourhood

I am not sure this is the best place to ask this kind of question about discussing the content of a paper. Anyway, here is my question: There is a famous paper from Physical Review E 69, 066138(2004) ...
0
votes
0answers
7 views

Cardinality of Time Sharing Random Variable for Multiple Access Channel

it is well know that capacity of MAC is \begin{align} R_1 \le I(X_1;Y|X_2,Q)\\ R_2 \le I(X_2;Y|X_1,Q)\\ R_1 \le I(X_1,X_2;Y|Q)\\ \end{align} where $|Q| \le 4$. How is the bound on $Q$ derived?I ...
1
vote
0answers
53 views

Information set of a linear code

I am trying to prove a couple of statements about information sets of linear codes, but i am having trouble with these proofs or i am not sure if i understand correct what i should prove. I would ...
0
votes
0answers
23 views

Huffman code with specific source

There is n-ary Huffman code. Source has the following relative frequencies of t symbols: 1, $n$, $n^2$, $n^3$, . . . , $n^{t−1}$, where $t = 1 + k(n − 1)$ for some positive integer $k$. I need to find ...
0
votes
1answer
14 views

Conditional entropy of repetition code over BSC

Consider the channel that takes in a bit, repeats it $k$ times, then sends the result over a binary symmetric channel with transition probability $p$. For example, if $0$ was sent over the channel ...
0
votes
2answers
50 views

Derivative of mutual information

Here is the definition of mutual information $I(X;Y) = \int_Y \int_X p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) } \; dx \,dy,$ where $x$ and ...
4
votes
0answers
109 views

Convergence of mutual information

Let $P_n (x,y)$ be a sequence of (cumulative) probability distributions defined on $\mathcal{X}\times \mathcal{Y}$ (of arbitrary cardinality), that weakly converges to $P(x,y)$: $$ P_n (x,y) ...
0
votes
1answer
38 views

what is the mutual information of three variables?

mutual information of tow variables is $\displaystyle\sum\sum p(x,y)\ln\frac{p(x,y)}{p(x)p(y)}$ what is the mutual information of three variables? is it $\displaystyle\sum\sum\sum ...
1
vote
1answer
104 views

Prove there exist a $p$ so that the inequality holds

I am stuck with the following problem. Given the Gaussian mixture distribution $f(\cdot)$ $$ f(x) = ...
1
vote
0answers
21 views

Minimum of an Entropy based function

This question is a small part of a bigger problem I am working on. Let $h(p)$ be the binary entropy function. That is, for $p \in (0,1)$ $$h(p) = -p\log_2(p) - (1-p)\log_2(1-p)$$ Define the ...
0
votes
0answers
37 views

How to compute the topological entropy of a permutation?

I have a permutation, say as ${4,1,7,2,3,5,6}$, given by its induced matrix. According to this paper (Proposition 11 on p. 82), To compute its topological entropy, one can compute the ...
0
votes
1answer
29 views

What is the meaning of E and d in this formula?

I am trying to learn the information bottleneck method. On slide 15, they give this equation. I think I understand that X is a random variable (but do not understand the meaning of the exponent, n). I ...
1
vote
0answers
69 views

optimization problem gaussian maximizes entropy

Let $X_1, X_2, Z_1$ be random variables and define $$Y=aX_1+bX_2+Z_1$$ I have the following optimization problem of difference of entropies, $$f=\max_{p(x_1x_2)} h(Y) - h(Y|X_2)= \max_{p(x_1,x_2)} ...
1
vote
0answers
25 views

Mutual information between two Gaussian distribution

Suppose we have two variables $x_i$ and $x_j$ with covariance matrices $P_i$ and $P_j$ and cross-covariance $P_{ij}$. I'd like to find the mutual information on them. From reverse engineering of some ...
1
vote
0answers
21 views

KSE and Shannon entropy

Is there a theoretical connection between Kolmogorov-Sinai and Shannon entropies? What is it?
1
vote
1answer
40 views

How many points does it take to identify a low-order polynomial in $\mathbb{Z}_N$?

I want to split the Bush's Baked Beans recipe into $M$ parts so that any set of $N<M$ people can reconstruct the recipe, but with the following constraints: Each person knows only a yes or no ...
0
votes
1answer
37 views

Probability density function of entropy of a gaussian variable

I have a problem finding the probability density function of entropy of a normally distributed sample. It is known that the entropy of a gaussian variable $X$ equals $H=h(X)={1\over2}\log(2\pi ...
1
vote
0answers
25 views

Bound the entropy knowing the largest denominator

The problem comes up from considering sampling from a discrete set of $n$ items with integer weights. The $i$th item has weight $w_i$, the probability getting chosen is $w_i/\sum_j w_j$. Certainly the ...
4
votes
2answers
48 views

Determinant of Fisher information

In information geometry, the determinant of the Fisher information matrix is a natural volume form on a statistical manifold, so it has a nice geometrical interpretation. But what is it in ...
0
votes
1answer
32 views

$H(X\mid Y_1, Y_2) \leq H(X\mid Y_1)?$ (Conditional Entropy with conditioning on multiple RVs)

In short, my question is whether the "conditioning reduces entropy" maxim is also true when conditioning on one random variable as compared to conditioning on two: $$H(X\mid Y_1, Y_2) \leq H(X\mid ...
0
votes
1answer
57 views

Is it possible to study information theory while studying a first course on probability?

I'm currently taking a course on intro to probability. The course is not mathematically rigorous and does not invoke theorems from real analysis, etc. The course covers all the way from basic ...
2
votes
1answer
28 views

Self-information, one event half as likely than another event conveys twice the amount of information?

I was reading the following: "If one event is half as likely as another, then learning about the former event shouldconvey twice as much information as the latter" I know it should be easy to ...
0
votes
1answer
50 views

How is this paper using probability notation?

I am trying to understand this paper about documents and sentences. At the end of page three, they say: Let g(wi, wj ) be the distance between two events (1 if in the same sentence, 2 in neighboring, ...
0
votes
1answer
38 views

Erasure Codes with Simplex Locality

In here, theorem $1.1$. there is this line 'Since $G$ has full rank it is possible to enlarge $N$ to a set $N^{'}$ ... exactly $k-1$. Note that the enlargement operation ... any of the leaders' in the ...
0
votes
0answers
24 views

Mathematics branch concerned with availability of information

Is there a branch of mathematics that study about availability of information? For example, if I want to search for something on the internet, is there a branch of mathematics that can predict how ...
-1
votes
1answer
46 views

Analysis of Kullback-Leibler divergence

Let us consider the following two probability distributions ...
1
vote
2answers
110 views

Upper bound on the entropy of a sum two random variables

Let $X$ be a random variable such that $|X| \leq A$ almost surely, for some $A > 0$. Let $Z$ be independent of $X$ such that $Z \sim {\cal N}(0, N)$. My question is: How large can the entropy ...
0
votes
1answer
24 views

mutual information of coupled variables

I have been looking for a method for evaluating the mutual information between a combination of source variables, $X_0, X_1$ and a target variable, $Y$. $$I(Y;X_0,X_1)$$ When I look on wikipedia's ...
1
vote
2answers
49 views

Ways to code two arbitrary binary strings into one without loss of information, and relevant bounds

If the title was not clear, I'm examining methods of taking two binary strings as input and outputting one binary string in such a way that the two original strings can be extracted from the output, ...
2
votes
1answer
47 views

partition with infinite entropy?

Let $P$ be an infinite partition of the interval $[0,1]$. Let $P$ have elements $I_i$ which has Lebesgue measure $m(I_i)$. Then the entropy of $P$ is defined by $\sum_i -m(I_i)\log m(I_i)$. Can this ...
1
vote
0answers
17 views

Mutual information staying constant under composition of channels

Consider the following scenario: one has 2 communication channels $C_1$ and $C_2$. Let $p_0(x)$ be some arbitrary but fixed input probability distribution. The mutual information between the input ...
0
votes
0answers
49 views

Special case of Kullback-Leibler additivity

I have three random variables $X,Y,Z$. If $(X,Z)$ are an independent pair and $(Y,Z)$ are an independent pair, then the additive property of the Kullback-Leibler divergence says $K(X,Z|Y,Z) = K(X|Y) ...
2
votes
1answer
23 views

Confusion about non-negative mutual information

The formula I was given for calculating information for a specific stimulus $s_x$ is: $$I(R,s_x) = \sum_i p(r_i|s_x) \log_2{p(r_i|s_x)\over p(r_i)} $$ It was also said that information is always ...