5
$\begingroup$

entropy (information content) is defined as:

$$ H(X) = \sum_{i} {\mathrm{P}(x_i)\,\mathrm{I}(x_i)} = -\sum_{i} {\mathrm{P}(x_i) \log_b \mathrm{P}(x_i)} $$

This allows to calculate the entropy of a random variable given its probability distribution.

But, what if I have a set of scalar samples and I want to calculate their entropy? In this case the probability density function is not available, but maybe there is a formula to get an approximation (as in the sample mean)? Does it have a name?

$\endgroup$
1

1 Answer 1

Reset to default
3
$\begingroup$

The most natural (and almost trivial) way to estimate (not calculate) the probabilities is just counting: $$\hat{p_i}=\frac{n_i}{N}$$ where $p_i$ is the probabilty of symbol $i$, $\hat{p_i}$ its estimator, $n_i$ the counting of ocurrences of symbol $i$, and $n$ the total of samples. Then you plug this estimator into the entropy formula.

However, this might not be a fair estimator of the entropy rate of your source, because it does not take into account the dependencies between succesive symbols. It only makes sense if the source emits independent symbols - or if you are only interested in the marginal entropies (and provided that your source is stationary - ergodic, actually).

$\endgroup$
2
  • $\begingroup$ thanks for your answer, but the samples are not discrete: they are bivariate real values (coordinates) $\endgroup$
    – fstab
    Jul 24, 2015 at 19:18
  • $\begingroup$ Then you shouldn't refer to that definition of entropy, that's for discrete variables $\endgroup$
    – leonbloy
    Jul 25, 2015 at 1:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .