4
votes
0answers
49 views

Decrease of entropy when iterating a random discrete function

Let $m$ be a positive integer. Let $S$ be the set of non-negative integers $x$ less than $m$, with $|S|=m$. Let $X_0$ be the discrete uniform distribution over $S$, with $P(x)=\begin{cases} 1/m & ...
0
votes
1answer
36 views

Wrong result from LLR using Dunning Entropy method

I'm trying to use Dunning's method of calculating LLR to compare word instances between two fulltext indexes. His method uses entropy as part of the calculation. Dunning's blog post: ...
2
votes
3answers
217 views

Is Standard Deviation the same as Entropy?

We know that standard deviation (SD) represents the level of dispersion of a distribution. Thus a distribution with only one value (e.g., 1,1,1,1) has SD equals to zero. Similarly, such a distribution ...
0
votes
3answers
77 views

Roll a dice, ignore result if result is maximum value

Let's say I have a six-faced dice, but I only want results between $1$ and $5$. One way to do that would be to roll a ten-faced dice and divide the result by two ($1-2$ becomes $1$, $3-4$ becomes ...
3
votes
2answers
121 views

Estimating the entropy

Given a discrete random variable $X$, I would like to estimate the entropy of $Y=f(X)$ by sampling. I can sample uniformly from $X$. The samples are just random vectors of length $n$ where the entries ...
1
vote
1answer
93 views

What is the maximum entropy distribution for a continuous random variable on $[0,\infty)$ with given mean and variance?

I know that for a given logmean and logstdev its the lognormal, but what about where we directly specify the mean and variance? The above seems to depend on the log-transformation to the maxent for ...
0
votes
2answers
49 views

Can infinite random sequences be asymptotically compressed?

A number $0.5<p<1$ is chosen at random and given to two people A and B whom are allowed to communicate before beeing separated. A is then given a sequence S of N random bits where each bit has ...
2
votes
0answers
134 views

Non-zero Conditional Differential Entropy between a random variable and a function of it

Let two continuous random variables, where the one is a function of the other: $X\, $ and $\, Y=g\left(X\right)$. Their mutual information is defined as ...
1
vote
0answers
183 views

Mutual Information of Correlated Bivariate Uniform Distribution

We have correlated bivariate uniform distribution, where X and Y have a correlation coefficient $\rho$ and they uniformly distributed in the following rectangle. What is the mutual information of $X$ ...
2
votes
1answer
54 views

Parameter optimization in probabilistic models

Task: Suppose we model a variable $y = Wx + \mu$ as a linear transformation of $x$ plus some Gaussian noise $\mu\sim\mathcal N(0,\sigma I)$. Our aim is to minimize the estimation error of $x$ given ...
0
votes
1answer
96 views

Test for randomness

I'm trying to write a program to compute a metric for the entropy in files to determine a probability that the file is compressed or encrypted. Compressed and encrypted files have very, very, very ...
0
votes
1answer
222 views

Relative Entropy given two non-equivalent sets

I am trying to calculate the relative entropy given two collections and have a question regarding some issues. Supposed we have two sets, $Real$ and $Calculated$, and their respective probability ...