The science of compressing and communicating information. It is a branch of applied mathematics and electrical engineering. Though originally the focus was on digital communications and computing, it now finds wide use in biology, physics and other sciences.
0
votes
1answer
216 views
Relation between Kraft's inequality and optimal entropy encoding codes
Given a binary code that verifies Kraft's inequality can I state that this code is optimal?
I know that optimal codes verify this inequality, like so $\sum\limits_{i=1}^{M} 2^{-\ell_i} \leq 1$ where ...
1
vote
1answer
151 views
Uses needed to break a substitution cipher (alphabet of 5 symbols, normalized entropy of the source of 2)
I'm studying for an information theory exam, maybe some of you can help me here with an exercise about cryptography.
I'm trying to understand how to calculate the minimum needed uses to break a ...
3
votes
1answer
208 views
Entropy of $X =\{1,2,\ldots,\infty\}$ with the probability of $\{1/2^1,1/2^2,\ldots,1/2^\infty\}$?
I'm studing for an information theory exam, maybe some of you can help me here with an exercise.
What's the entropy of $X$ as $\{1,2,\ldots,n\}$ ($n$=infinity) where the probabilities are $P \{1/2^1, ...
8
votes
2answers
358 views
What is the relationship between the Boltzmann distribution and information theory?
I'm reading a paper on Boltzmann machines (a type of neural network in Machine Learning), and it mentions that "The Boltzmann distribution has some beautiful mathematical properties and it is ...
1
vote
3answers
281 views
Is it wrong to use Binary Vector data in Cosine Similarity?
I am doing Information Retrieval using Cosine Similarity.
My data is binary vector.
Since most of all reference I read is using non-binary vector (non-binary matrix) data, I am wondering if it is ...
3
votes
1answer
192 views
Minimum Mean Square Error (MMSE) and Mutual Information (I)
Consider this setting:
$Y=X+N$
where $N$ is a Gaussian standard random variable and $X$ is another arbitrarily distributed r.v. You can think of this $X$ as a message being transmitted over an AWGN ...
20
votes
2answers
972 views
I'm not sure about this inequality (how to prove or disprove it?)
For $a_1,...,a_n,b_1,...,b_n>0,\quad$ define $a:=\sum a_i,\ b:=\sum b_i,\ s:=\sum \sqrt{a_ib_i}$.
Is the following inequality true?:
$${\frac{\Bigl(\prod a_i^{a_i}\Bigr)^\frac1a}a \cdot ...
0
votes
1answer
161 views
Relative Entropy given two non-equivalent sets
I am trying to calculate the relative entropy given two collections and have a question regarding some issues.
Supposed we have two sets, $Real$ and $Calculated$, and their respective probability ...
-1
votes
1answer
183 views
Entropy and Information Theory
Consider a binary message in which $0$ has has probability $1/3$ and $1$ has probability $2/3$. What value of $H$ should be assign?
I know that you split up $1$ into two messages $1a$ and $1b$. Then ...
3
votes
1answer
346 views
Kullback-Leibler divergence based kernel
I'm looking to paper "A Kullback-Leibler Divergence Based Kernel for SVM Classification in Multimedia Applications". Author suggest to use kernel function for two distributions $p$ and $q$: $k(p,q)= ...
3
votes
1answer
182 views
Similarity between Entropy of information source & Expectation of a random varible
While I was watching a lecture on Information theory, I found that entropy of an information source is the average amount of information that it provides in terms of bits (or nats, decits or ...
2
votes
2answers
636 views
Shannon's formula
Shannon formally deļ¬ned the amount of information in a message as a function of the probability of the occurrence of each possible message[1]. Given a universe of messages $\mathbf{M} = \{ ...
2
votes
2answers
689 views
Using KKT conditions to maximize function
The goal is to maximize the following function:
\begin{align}
K_p(q) = q\log \frac{q}{p} + (1-q)\log \frac{1-q}{1-p}
\end{align}
where
\begin{align}
0 \leq q \leq 1
\end{align}
and $p \in (0,0.5)$ and ...
3
votes
1answer
197 views
Entropy of generatable(?) structures
In many places I see the entropy definition as:
$H(X) = \sum_{i=1}^n {p(x_i)\,I(x_i)} = -\sum_{i=1}^n {p(x_i) \log_b p(x_i)}$
In Wikipedia I saw: $H(X) = \operatorname{E}(I(X))$ where E is ...
1
vote
1answer
125 views
Can the mutual information of a “cell” be negative?
Please forgive me if this is not the right Stack Exchange (I also posted it at Cross Validated and Theoretical Computer Science). Please also forgive me for inventing terms.
For discrete random ...
5
votes
2answers
233 views
Theoretical basis for overfitting
There are many examples in which making more "precise" predictions gives worse performance (e.g. Runge's phenomenon). My professor implied that there was a sound basis for choosing "simple" functions ...
