The entropy tag has no wiki summary.
0
votes
1answer
35 views
Solving exponential equations for maxent with opensource applications or coding
I am working on maximum entropy. And I have a problem solving some equations.
Look at these 2 equations:
$\frac{(2*e^{-a*2-b*2^2}) +(6*e^{-a*6-b*6^2}) +(10*e^{-a*10-b*10^2}) ...
1
vote
1answer
20 views
How Entropy scales with sample size
For a discrete probability distribution, the entropy is defined as:
$$H(p) = \sum_i p(x_i) \log(p(x_i))$$
I'm trying to use the entropy as a measure of how "flat / noisy" vs. "peaked" a distribution ...
3
votes
1answer
163 views
Notation of cross entropy
I have a question regarding a notation that seems to be very usual.
For starters, cross entropy is defined by:
\begin{align}H(X, q) &= H(X) + D(p||q) \\
& =-\sum_x p(x)\log_2 q(x)\end{align}
...
0
votes
1answer
22 views
Help with a solution about entropy!
I'm triying to solve this question question, but I don't understand why
$$ -\log_2 (2\pi n pq)/(2n) $$ transforms into $$O(\ln n/ n)$$
can you help me please :( !!
1
vote
1answer
17 views
Entropy contribution from variable length segment of a sequence
If I have a sequence which is comprised of one of 10 prefixes, one of 5 suffixes and a variable length middle, how do I compute the entropy of the sequence?
Using Shannon-Entropy $Hs= ...
1
vote
1answer
33 views
Constructing Distribution By Coin Flipping
I am interested in any example of construction distribution by coin flipping.
Actually I want to show the process of construction a random variable $X$ with distribution $(p_1,...,p_n)$ by coin ...
0
votes
0answers
38 views
Intregral of exponential of Shannon Entropy Function
Here I am going to ask a similar question as rde asked , that is what is the integral of exponential of entropy function. That is what is the value of
$F[H(x)]=\int_{-1}^{+1} e^{ikH(f(x^2))} dx$
...
4
votes
2answers
78 views
Are there logarithm functions for arbitrary rings?
The logarithm function for $\mathbb{R}$ satisfies $\log xy = \log x + \log y$ whenever both $\log x$ and $\log y$ are defined.
Are their conditions for a ring $R$ which guarantee the existence of a ...
4
votes
0answers
67 views
A combinatorial problem.
Let be $(X, \mathbb{A}, \mu)$ a measure space, a partition of $ X $ is a disjoint family $\xi=\{P_1,\ldots,P_k \}$ of measurable sets such tath $\bigcup P_i=X\pmod0).$
If $\xi=\{P_1,\ldots,P_k ...
0
votes
0answers
84 views
Replace a continuous probability distribution with a discrete one
Say one wants to fit a curve $f(x)$ to a set of noisy data points $(x_i, y_i)$. If the error for each point $y_i$ is assumed to be normally distributed with variance $\sigma_i^2$, one wants to find ...
1
vote
1answer
26 views
Entropy vs predictability vs encodability
Imagine there's a guessing game where a series of binary symbols are presented and a human must decide quickly if the symbol is the same as the previous or different. There's a property of the ...
1
vote
1answer
28 views
Shannon inequalities
I have some difficulties in showing the relationship between mutual information
$I(X; Y |Z)$ and $I(X; Y)$?
What is larger?
1
vote
1answer
50 views
Conditional entropy of sum of random variables
How can be proven that for random variables $A$ and $B$, and $C = A + B$,
$$H(C\mid A) = H(B\mid A).$$
Also, would it be possible to determine if $H(C)$ would be greater than $H(A)$?
2
votes
0answers
29 views
Computing Relative entropy?
I am doing a project for my CS class and I was wondering if the following would work.
I have 50 different people who have rated the same 50 books. The rating system is as follows:
negative 5 = hate ...
1
vote
0answers
29 views
convergence of discrete random variables with finite entropy
Let $Z$ be the set of discrete random variables on some probability space. Define the quantity $d(X_1,X_2)=h(X_1 \mid X_2)+h(X_2 \mid X_1)$ between two random variables $X_1, X_2 \in Z$. For $X \in Z$ ...
3
votes
0answers
71 views
Which takes more energy: Shuffling a sorted deck or sorting a shuffled one?
You have an array of length $n$ containing $n$ distinct elements. You have access to a comparator on the elements (a black-box function that takes $a$ and $b$ and returns true if $a < b$, false ...
0
votes
0answers
29 views
convexity of the product of two entropy-like functions
Consider the functions $T_p(q)= \sum_i q_i^p$, where p>1 and q is a finite-dimensional vector satisfying $\sum_i q_i = 1, q_i >0$ (ie, a probability mass function). In information-theoretic terms, ...
2
votes
1answer
24 views
Do the pth powers of $p$-norms define the same partial ordering on the set of all probability distributions for all $p>1$?
Consider the $p$-th power of the Schatten $p$-norm $||q||_p$ of a probability distribution $q$ , ie, the function $\sum_j q_j^p$, where $\sum_j q_j = 1$ and $q_j \geq 0$. For fixed $q$ and $p>1$ ...
0
votes
1answer
32 views
i.i.d binary random variable question
Suppose there are i.i.d. binary random variables $X_i \sim X$ with distribution $P(X=1) = 0.75$ and $P(X=0) = 0.25$
i) For $n=5$ and $e=0.1$, which sequences fall in the typical set $A_e^n$? What is ...
1
vote
0answers
75 views
Why is the negative entropy Lipschitz with respect to the $1$-norm (Over)?
Let $\left\|x \right\| = \sum_{i=1}^{i=n}\left|x^i\right|$ and $d\left(x\right)=\sum_{i=1}^{i=n}x^i\ln x^i$ where $x\in R^n $ and $ \sum_{i=1}^{i=n}x^i=1$
How to prove:
For all $x, x'$, $$\left| ...
1
vote
3answers
175 views
Minimum number of bits required to store the order of a deck of cards
Assume I have a shuffled deck of cards (52 cards, all normal, no jokers) I'd like to record the order in my computer in such a way that the ordering requires the least bits (I'm not counting look up ...
0
votes
1answer
64 views
Approximating probability of success of Bernoulli trials using Kullback–Leibler divergence
In "Probabilistic Graphical Models" book by Daphne Koller and Nir Friedman they have the following approximation of probability of r successful outcomes of N Bernoulli trials:
$P(S_N=r)\approx ...
2
votes
1answer
91 views
Definition of the Entropy
I have a question regarding definition of entropy by expected value of the random variable $\log \frac{1}{p(X)}$:
$H(X) = E \log \frac{1}{p(X)}$,
where $X$ is drawn accordingly to the probability ...
2
votes
1answer
81 views
Definition of entropy of an ergodic measure
I'm reading a paper in which it is stated that
The entropy of an ergodic measure is defined as
$$\lim_{n \to \infty} -\frac{1}{n} \sum_{|w|=n} \mu[w] \log \mu[w].\tag{1} \label{eq:1}$$
Here ...
1
vote
1answer
95 views
Bits in a coin-toss experiment
This is not homework but an actual problem.
We flip a fair coin ten times. This gives A$_1$ to A$_{10}$. Each coin toss = 10 bits. We flip another fair coin ten times. This gives B$_1$ to ...
3
votes
1answer
133 views
Question about entropy
Let $(X,A,\nu)$ be a probability space and $T\colon X\to X$ a measure preserving transformation $\nu$. Take a measurable partition $P=\{P_0,\dots,P_{k-1}\}$. Let$I$ be a set of all possible ...
0
votes
0answers
67 views
Proof for the upper bound on entropy $H(S)$?
I was trying to prove the upper bound on $H(S)$ using the inequalities $\ln(x)\le(x-1)$ and $\ln(1/x)\ge(1-x)$ for independent and memory less source symbols $s_1,\dots,s_q$ .
I am trying to prove ...
1
vote
1answer
57 views
About the differential entropies of well-known continuous distributions
Assume that the continuous random variable $X$ has a distribution (in a closed form expression) with differential entropy $h(X)$.
Q) Then, is it true for any continuous distribution that the ...
0
votes
1answer
57 views
Entropy Problem: mutual information
I have a problem about entropy and mutual information that I have attempted, but would like feedback on.
30% Boas
20% Anaconda
50% Cobra
Half of the Cobras were medium sized, and the other half were ...
5
votes
1answer
141 views
Pinsker $\sigma$ Algebra
Let $(X,A,\nu)$ be a probability space and $T:X\to X$ a measure-preserving transformation.
The Pinsker sigma algebra is defined as the lower sigma algebra that contains all partition P of measurable ...
0
votes
0answers
54 views
Reversible and irreversible gates
Is it possible to make any gate reversible merely by retaining the input bits in the
output and introducing ancilla bits as necessary? That is, given an irreversible
gate with k inputs and l outputs, ...
0
votes
0answers
33 views
entropy change relation to the number of lost bits
Can we use entropy change value to define the number of bits of information that are lost for and, or, xor gates?
what is the number of bits lost due and, or, xor gates?
Thanks much in advance!!!
0
votes
1answer
89 views
How to prove the following entropy formula?
Could anyone show me a proof or redirect to a source where the following entropy equation is proved? =)
Thank you!
3
votes
1answer
105 views
convergence of entropy and sigma-fields
This question is related to this one.
Let $(X_1, X_2, \ldots)$ be a sequence of random variables such that each $X_n$ takes its values in a finite space, say $\{0,1\}$, and the $\sigma$-field ...
6
votes
2answers
137 views
Can the entropy of a random variable with countably many outcomes be infinite?
Consider a random variable $X$ taking values over $\mathbb{N}$. Let $\mathbb{P}(X = i) = p_i$ for $i \in \mathbb{N}$. The entropy of $X$ is defined by
$$H(X) = \sum_i -p_i \log p_i.$$
Is it possible ...
3
votes
2answers
73 views
Calculate entropy with n values
I trying to solve a quiz that asks the following.
The variable $X$ can be the values $1,2,3,...,n$ with the probabilities $\frac{1}{2^1}, \frac{1}{2^2},\frac{1}{2^3},...,\frac{1}{2^n}$
How can I ...
0
votes
1answer
168 views
equivalence between uniform and normal distribution
The principle of insufficient reason says that all outcomes are equiprobable when we have no knowledge to guess otherwise. I understand this and that this corresponds to uniform distribution. However, ...
2
votes
1answer
499 views
Entropy of matrix
I am trying to understand entropy. From what I know we can get the entropy of a variable lets say X.
What i dont understand is how to calculate the entropy of a matrix say m*n. I thought if columns ...
20
votes
2answers
972 views
I'm not sure about this inequality (how to prove or disprove it?)
For $a_1,...,a_n,b_1,...,b_n>0,\quad$ define $a:=\sum a_i,\ b:=\sum b_i,\ s:=\sum \sqrt{a_ib_i}$.
Is the following inequality true?:
$${\frac{\Bigl(\prod a_i^{a_i}\Bigr)^\frac1a}a \cdot ...
1
vote
2answers
285 views
How are Huffman encoding and entropy related?
The inherent unpredictability, or randomness, of a probability distribution can be measured by the extent to which it is possible to compress data drawn from that distribution.
$$ \text{more ...
6
votes
0answers
50 views
Approximation of Shannon entropy by trigonometric functions
Define Shannon entropy by $$I(p) = -p \log_2 p$$ Numerical experimentation shows that
$\sin(\pi p)^{1-1/e}$ is a good approximation to $I(p) + I(1-p)$ on $[0,1],$ never differing by more than 3.3%. ...
5
votes
2answers
206 views
Inverse of binary entropy function for $0 \le x \le \frac{1}{2}$
I'm trying to find the inverse of $H_2(x) = -x \log_2 x - (1-x) \log_2 (1-x)$[1] subject to $0 \le x \le \frac{1}{2}$. This is for a computation, so an approximation is good enough.
My approach was ...
6
votes
1answer
455 views
Entropy of a binomial distribution
How do we get the functional form for the entropy of a binomial distribution? Do we use Stirling's approximation? According to Wikipedia, the entropy is $\frac1 2 \log_2 \big( 2\pi e\, np(1-p) \big) + ...
1
vote
1answer
106 views
Increasing entropy of random walk in regular graph
Let $P$ be a transition matrix of a random walk in an undirected regular graph $G$. Let $\pi$ be a distribution on $V(G)$. The Shannon entropy of $\pi$ is defined by
$$H(\pi)=-\sum_{v \in ...
1
vote
1answer
45 views
What is the CONCEPT when we speak of maximum entropy?
What is an intuitive interpretation of the concept of maximum entropy? I want to understand this concept better but what I'm finding is too "advanced" right now. Can anyone simplify it ... imagine I'm ...
0
votes
1answer
57 views
Entropy of a Binary Source with a random until first other result is given.
I was studying for an exam and i found an interesting exercise, but very very bad redacted.
A coin is thrown until the first face is found. Denote as X the number of throws required. And find:
a) ...
2
votes
1answer
43 views
Parameter optimization in probabilistic models
Task: Suppose we model a variable $y = Wx + \mu$ as a linear transformation of $x$ plus some Gaussian noise $\mu\sim\mathcal N(0,\sigma I)$. Our aim is to minimize the estimation error of $x$ given ...
1
vote
0answers
90 views
Mutual Information of Correlated Bivariate Uniform Distribution
We have correlated bivariate uniform distribution, where X and Y have a correlation coefficient $\rho$ and they uniformly distributed in the following rectangle. What is the mutual information of $X$ ...
0
votes
2answers
104 views
Prove that entropy is maximized when probability is $1/n$
How can be proven that the entropy of a dice roll is maximized when the probability of each of its $6$ faces is equal, $1/6$?
0
votes
1answer
186 views
Kullback-Leibler distance between 2 probability distributions
Can I determine the Kullback-Leibler distance
$$
D_{\mathrm{KL}}(P\parallel Q)=\sum_{i}\ln\left(\frac{P(i)}{Q(i)}\right) P(i)
$$
between the following probability distributions?
...
