1
$\begingroup$

I need to calculate the following enter image description here

which is Shannon Entropy formula only using simple functions like log or squareroot or I don't know, just make it simple enough for a guy that does understand only grammar school math, if that is possible, so I don't understand this at all. For starters I don't even know what to do with the "big E" in the beginning of each part of this formula, that is how I "understand" math :(

$\endgroup$
  • $\begingroup$ The big E (Sigma, $\Sigma$) means sum over all $i$. $\log(1/x) = -\log(x)$ is a well known identity of the logarithm. Would you know how to calculate the entropy given the $x_i$? $\endgroup$ – mlainz Apr 30 '16 at 4:23
  • $\begingroup$ Or do you want to understand why do we use that formula? $\endgroup$ – mlainz Apr 30 '16 at 4:25
  • $\begingroup$ The formula looks a little bit scary, but it's actually very simple. $\endgroup$ – mlainz Apr 30 '16 at 4:38
1
$\begingroup$

For example, let $b = 2$. You have 3 events $x_1, x_2, x_3$,

$$p(x_1) = 0.2\\ p(x_2) = 0.5 p(x_3) = 0.3$$

Then $$H = -\sum_i p(x_i) \log_2(p(x_i)) = - p(x_1) \log_2(p(x_1))- p(x_2) \log_2(p(x_2))- p(x_3) \log_2(p(x_3)) =\\ 0.46 + 0.50 +  0.52 = 1.48 $$

Perhaps, if you tell me in which context you found that formula, I could give you a more meaningful example.

$\endgroup$
  • $\begingroup$ Ok. For binary systems you always use $b = 2$. It might be worth to check wikipedia page on password strength $\endgroup$ – mlainz Apr 30 '16 at 4:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.