0
votes
1answer
13 views

Normalization of data in decision tree

After reading through a few references, I have come to know that for machine learning in general, it is necessary to normalize features so that no features are arbitrarily large ($centering$) and all ...
5
votes
0answers
54 views

Decrease of entropy when iterating a random discrete function

Let $m$ be a positive integer. Let $S$ be the set of non-negative integers $x$ less than $m$, with $|S|=m$. Let $X_0$ be the discrete uniform distribution over $S$, with $P(x)=\begin{cases} 1/m & ...
0
votes
2answers
48 views

Wrong result from LLR using Dunning Entropy method

I'm trying to use Dunning's method of calculating LLR to compare word instances between two fulltext indexes. His method uses entropy as part of the calculation. Dunning's blog post: ...
2
votes
3answers
280 views

Is Standard Deviation the same as Entropy?

We know that standard deviation (SD) represents the level of dispersion of a distribution. Thus a distribution with only one value (e.g., 1,1,1,1) has SD equals to zero. Similarly, such a distribution ...
0
votes
3answers
79 views

Roll a dice, ignore result if result is maximum value

Let's say I have a six-faced dice, but I only want results between $1$ and $5$. One way to do that would be to roll a ten-faced dice and divide the result by two ($1-2$ becomes $1$, $3-4$ becomes ...
3
votes
3answers
143 views

Estimating the entropy

Given a discrete random variable $X$, I would like to estimate the entropy of $Y=f(X)$ by sampling. I can sample uniformly from $X$. The samples are just random vectors of length $n$ where the entries ...
1
vote
1answer
110 views

What is the maximum entropy distribution for a continuous random variable on $[0,\infty)$ with given mean and variance?

I know that for a given logmean and logstdev its the lognormal, but what about where we directly specify the mean and variance? The above seems to depend on the log-transformation to the maxent for ...
0
votes
2answers
50 views

Can infinite random sequences be asymptotically compressed?

A number $0.5<p<1$ is chosen at random and given to two people A and B whom are allowed to communicate before beeing separated. A is then given a sequence S of N random bits where each bit has ...
3
votes
0answers
145 views

Non-zero Conditional Differential Entropy between a random variable and a function of it

Let two continuous random variables, where the one is a function of the other: $X\, $ and $\, Y=g\left(X\right)$. Their mutual information is defined as ...
1
vote
0answers
192 views

Mutual Information of Correlated Bivariate Uniform Distribution

We have correlated bivariate uniform distribution, where X and Y have a correlation coefficient $\rho$ and they uniformly distributed in the following rectangle. What is the mutual information of $X$ ...
2
votes
1answer
54 views

Parameter optimization in probabilistic models

Task: Suppose we model a variable $y = Wx + \mu$ as a linear transformation of $x$ plus some Gaussian noise $\mu\sim\mathcal N(0,\sigma I)$. Our aim is to minimize the estimation error of $x$ given ...
0
votes
1answer
100 views

Test for randomness

I'm trying to write a program to compute a metric for the entropy in files to determine a probability that the file is compressed or encrypted. Compressed and encrypted files have very, very, very ...
0
votes
1answer
232 views

Relative Entropy given two non-equivalent sets

I am trying to calculate the relative entropy given two collections and have a question regarding some issues. Supposed we have two sets, $Real$ and $Calculated$, and their respective probability ...