# Tagged Questions

The science of compressing and communicating information. It is a branch of applied mathematics and electrical engineering. Though originally the focus was on digital communications and computing, it now finds wide use in biology, physics and other sciences.

43 views

### Fisher Expected Information for a Gaussian Process model

Suppose I have a two dimensional Gaussian process model (GP), defined by a squared exponential correlation function s.t: $$R(x_{i},x_{j}) = \exp\left(-\frac{|x_{i} - x_{j}|^2}{2}\right).$$ I am ...
70 views

### sufficient and necessary condition for equality between conditional mutual information and unconditional one.

Suppose $X, Y, Z$ are three discrete random variables. Is there a good sufficient and necessary condition for $I(X;Y|Z) = I(X;Y)$? Usually the LHS can be bigger or smaller than the RHS, but if Z is ...
34 views

I'm trying to understand the proof that $I(X;Y)$ is convex in conditional distribution $p(y \mid x)$ - from Elements of Information Theory by Cover & Thomas, theorem 2.7.4. In the proof we fix $p(... 2answers 85 views ### Question regarding the Entropy of a probability mass function I assume that the entropy,$E$, of a probability mass function (pmf),$p(X)$, of a discrete random variable,$X, is computed as: \begin{align}\mathbb{E}(p(X)) &= -p(X = x_1)\log[p(X = x_1)]-p(... 1answer 91 views ### Relation between Genetic Algorithm and Information theory Can anyone suggest me some references (papers, books, lecture notes) on the relation between GA and Information theory? 1answer 38 views ### Convexity of I(X;Y): why H(Y) convex in p(y) \Rightarrow H(Y) convex in p(x) I would like to understand the proof that mutual information I(X;Y) is concave in p(x) - as presented in Elements of Information Theory by Cover & Thomas, theorem 2.7.4. Here's the proof from ... 1answer 37 views ### conditional entropies for identical distributions Let me say I have two distributions X and Y which are identical, but they are not independent. Now if were to calculate the conditional entropies H(X|Y) and H(Y|X). Is calculating one joint ... 2answers 68 views ### Probability of inequality, Markov inequality application A bit of context: working on a problem about channel coding. Through a channel we are sending a random variable X_n, a code, and at the other side we see Y (both discrete). Then we perform an ... 0answers 41 views ### Doubt in derivation of a proof in Information Theory In my class we were trying to derive Shanon's Source Theorem, first by proving the equivalent form in a weaker version. The question is -Consider a biased coin with probability of heads p \geq \frac{... 1answer 65 views ### Standard deviation of a baised d-sided coin I know that that standard deviation of a noisy bit (a biased coin with probability distribution \{ p, 1-p \} ) is given by \sigma = \sqrt{p(1-p)} $$What is then a measure of the standard ... 2answers 75 views ### Strategy to find out how wires are connected There is a tube with 100 electrical wires that are not labeled. At side A of the tube, the terminal ends of the 100 electrical wires can be connected. It is possible to connect more than 2 ... 0answers 94 views ### Proving or disproving concavity of a function I want to prove that the following function is concave (as a part of another proof).$$f(p) = \max_{\begin{matrix}x,y\\0\le x \le 1\\0\le y \le 1 \\ x * y = p\end{matrix}} \lambda h(x) + \bar{\... 0answers 86 views ### Absolute value of difference between entropies (of two distributions) I have the following inequality for theL_1$distance between two distributions$Q$,$Q^n$on a finite set$B$: $$\|Q-Q^n|| < \frac{2|B|}{n}\leq \frac{C}{n} \leq \frac12$$ Assuming$C\geq2|B|$, ... 1answer 182 views ### Is Entropy = Information circular or trivial? I have seen several "maximum entropy distributions" used in the mathematical and statistical literature, often with the justification that they are "minimally informed" beyond the assumptions and data ... 2answers 229 views ### Is Information Theory Mathematics? When I read about Information Theory, for example on Wikipedia, I can never find statements that say if Information Theory is "real" Mathematics with underlying axioms, a notion of "proof beyond doubt"... 1answer 362 views ### What does “to first order in exponent” mean? I am studying information theory on "Elements of Invormation theory" (Cover Thomas). I cannot understand the meaning of "to first order in exponent" in the following theorem: ............................. 3answers 3k views ### How is logistic loss and cross-entropy related? I found that Kullback-Leibler loss, log-loss or cross-entropy is the same loss function. Is the logistic-loss function used in logistic regression equivalent to the cross-entropy function? If yes, can ... 2answers 62 views ### Channel code for multiple bit errors I've been exploring information theory out of personal interest and have a cursory understanding of Hamming Codes. From what I can tell, they're designed to exclusively detect the location of a single ... 1answer 102 views ### From Orthogonal vectors to Useful Bivector If we have set of orthogonal vectors (X) can we form a set of orthogonal bivectors from that set? I am trying to find if there is a way to get 'more information' from an orthogonal matrix by some ... 1answer 65 views ### Understanding an application of Entropy I'm struggling with the following exercise on entropy. Suppose that your friend Alice chooses a number$X$uniformly at random from$[1,n]$, which she writes down using$\log n$bits; you can assume ... 0answers 90 views ### Random Codebook Generation I do generate a random codebook$\mathcal{C}$by generating$2^{NC}$codewords$X^N=[X_1\;X_2\;\cdots\;X_N]$randomly and independently, each according to some distribution$p_{X^n}(x^n)=\Pi_{i=1}^n ...
Suppose $P_{Y|X}$ is a discrete memoryless channel with confusability graph $G$ and capacity $C = max_{P_X}I(X; Y )$. I want to prove the following relation: $\log{\alpha(G)}\le C$ where $\alpha(G)$ ...