leonbloy
Reputation
27,451
99/100 score
 16h comment Intuitive explanation of Shannon's source entropy in information / communications theory @SKM Yes, that's basically the first Shannon theorem. You need (on average) at least $H$ bits to encode a source (not a message!) of entropy $H$. en.wikipedia.org/wiki/Shannon%27s_source_coding_theorem (BTW: you expression "a source of length $N$" smells bad - a source has no length) 1d comment Intuitive explanation of Shannon's source entropy in information / communications theory You don't apply the Shannon entropy to "calculate the number of bits required to encode a message." You apply it to calculate (or rather bound) the average number of bits required to encode a source that emits random messages according to some known probability distribution. Shannon entropy is a probabilistic concept, and it requires a probabilistic model to be applied. 1d comment Why do we like sticking random variables into their own distributions? "without foresight, $f(X)$ seems like a strange object to consider". I agree. To compute the entropy we compute the expectation of a thing that involves the density itself, and that's rather unusual. I'm not sure on your conclusion that "$f(X)$ is extremely fundamental". Apr 29 comment Sending bits and parity bits over noisy channel What matters is the probability of getting the original bits right, for computing that you need to specify how you plan to use the parity bits to decode. Of course, in general (for a rational coder-decoder pair) adding redundacy (parity bits) should decrease the overall probabiblity of error. Apr 28 awarded Good Answer Apr 27 comment Conditional entropy under quantization @Bernhard I'm ok with that. But, then, the (Shannon) entropy $H(Y|X=x)$ with $Y$ continuous is $+\infty$ unless the density consists of Dirac deltas (i.e. unless $Y$ is discrete - informally speaking). Apr 26 comment Conditional entropy under quantization @Bernhard What is $H(Y|X)$ when $Y$ is continuous? It's a differential entropy or not? How do you define it? Apr 26 revised Computing mean of probability density function without integration added 63 characters in body Apr 26 answered Computing mean of probability density function without integration Apr 26 answered A fair coin is continually flipped until heads appears for the 10th time. Find the number of expected tails Apr 26 comment Computing mean of probability density function without integration Have you plotted it? Apr 22 comment Channel capacity of sum of symmetric channels "so they now have different input alphabets but the same output alphabet" That's not very clear for me. Could you elaborate? Apr 20 revised What is a probability that at 2nd turn you will pick green ball? added 8 characters in body Apr 18 comment number of ways to put N labelled balls in N labelled boxes so that labels don't match? This is known as derangement en.wikipedia.org/wiki/Derangement Apr 17 revised proof of upper bound on differential entropy of f(X) added 2 characters in body Apr 11 revised Is stating $\operatorname{E}(x) = 0, \operatorname{cov}(y,x)=0$, the same as stating $\operatorname E(y\,|\,x)=0$? added 28 characters in body Apr 11 revised Is stating $\operatorname{E}(x) = 0, \operatorname{cov}(y,x)=0$, the same as stating $\operatorname E(y\,|\,x)=0$? added 595 characters in body Apr 11 answered Is stating $\operatorname{E}(x) = 0, \operatorname{cov}(y,x)=0$, the same as stating $\operatorname E(y\,|\,x)=0$? Apr 11 comment Assumptions required for expected value of sum of products to equal zero. That's written using expected value (of $v$). Covariances have nothing to do here - if $x$ is a constant. Apr 11 answered Assumptions required for expected value of sum of products to equal zero.