Linked Questions

0
votes
3answers
137 views

How can they come up with the definition of entropy in information theory? [duplicate]

I have read some books about information theory but I don't have any ideas how can they find the definition of entropy? We have $$H(X)=-\sum_{x\in X}p(x)\, \text{log}\, p (x)$$ X is a discrete random ...
0
votes
1answer
99 views

Why do I find no clear answer for the log term in Shannon's theorem? [duplicate]

The explanations about Shannon's theorem speed up all of a sudden when they should tell us why he introduced the log term. They usually range from 'it's useful' to 'it's because there are two choices, ...
9
votes
4answers
2k views

Integral of odd function doesn't converge?

When I look up $$\int_{-1}^1 \dfrac{1}{x} dx$$ on Wolfram Alpha, it says it doesn't converge. While this is a sum of two diverging integrals, the two areas are clearly symmetric, and I'd assume the ...
4
votes
3answers
7k views

How is the formula of Shannon Entropy derived?

From this slide, it's said that the smallest possible number of bits per symbol is as the Shannon Entropy formula defined: I've read this post, and still not quite understand how is this formula ...
3
votes
5answers
1k views

Could the Riemann hypothesis be provably unprovable?

I don't know much about foundations and logic, so I ask forgiveness if my question is just plain stupid. Assume we have a statement of the form: There exist no $x\in X$ such that $P(x)$. where $X$...
4
votes
2answers
723 views

What does $-p \ln p$ mean if p is probability?

In statistical mechanics entropy is defined with the following relation: $$S=-k_B\sum_{i=1}^N p_i\ln p_i,$$ where $p_i$ is probability of occupying $i$th state, and $N$ is number of accessible ...
13
votes
1answer
646 views

What is the most extreme set 4 or 5 nontransitive n-sided dice?

A set of nontransitive dice is a set of dice whose face numbers are such that the relation "is more likely to roll a higher number than" is not transitive. (See wikipedia) For some sets, the ...
8
votes
1answer
348 views

Understanding information entropy

I have tried very very hard to get a good intuitive grasp on entropy. I read all the information theoretic intuitions on the web, but I have not exactly been able to understand what information is, ...
4
votes
2answers
235 views

Could someone show me a simple example of something being proved unprovable?

Could someone show me a simple example of something being proved unprovable? Pretty much what the title says, I want to understand a proof of some statement being proved unprovable. E: Please read ...
4
votes
2answers
220 views

What is the Geometric interpretation of $i^i$?

We know that $i^i$ is real. But how to explain it geometrically maybe in terms of rotation. like we can explain geometrically multiplication of two complex numbers and so on. Can someone show me a ...