Quantity of information in a neural network

Question

If I have a binary sequence that describes the trigger pattern of a neuron such that each 0 or 1 represents an interval of 10ms and each value 1 represents an "activation" of the neuron and each 0 represents "not activation" like this

$${1001101011 1010111010 0000001101 0100010001 1110110010 0100001001 \ ,}$$

then what is the information transmited, let's say, in 50ms?

What I did was calculating frequences and then the Shannon entropy with this probabilities, and then the passage from 10ms to another interval is just a multiplication. Is that right?

Answer 1

What I did was calculating frequences and then the Shannon entropy with this probabilities, and then the passage from 10ms to another interval is just a multiplication. Is that right?

What you did was an estimation of the marginal probabilities assuming binary values (states of activation or no activation) are independent. That is a wild (probably wrong) assumption to make.

It's not simple to estimate the entropy of a source given just a realization of it. In fact, it's practically hopeless. For one thing, we need at least to assume ergodicity.

In your case, you could try to improve the entropy estimator by estimating the bit-to-bit transition probabilities (which would amount to assume a first order Markov chain model) and computing the resulting entropy rate, but, again, this probably would not be enough to capture all the probabilitic structure of your source.

A simple empirical recipe to estimate the entropy of a source, if you have a quite long sample, is : compress it.