# Tagged Questions

This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

27 views

33 views

24 views

25 views

### How to test quality of probability estimates?

I have a Markov chain model which produces a probability distribution for absorption in 4 possible absorbing states. I.e. the model estimates the probability distribution for a discrete random ...
17 views

### Channel capacity of sum of symmetric channels

I've got a channel matrix $P$ of the form $\begin{bmatrix} Q \\ R \end{bmatrix}$ where $Q,R$ are channel matrices of symmetric channels, so they now have different input alphabets but the ...
28 views

### Basic Entropy Inequality and Identity question

This is a solution to a problem I am working on: \begin{aligned} H(X|Y) + H(Y|Z) &\ge^? H(X|Y, Z) + H(Y|Z) \\ &=^\text{?}H(X,Y |Z) \\ &= H(X|Z) + H(Y|X, Z)\\ &\ge H(X|...
21 views

### Entropy of degenerate multivariate normal

After having read the Wikipedia entry on the degenerate case of the multivariate normal distribution: https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case my question is: ...
26 views

### Calculating Entropy and Information Gain of a Variable

I have the following values for two random variables. I need to compute the following values: a. H(Y) b. H(Y|X) c. and finally IG(Y|X) I will show what I have calculated so far. a. H(Y) = -(.5*...
17 views

### Estimation for entropy

Let $T\colon X\to X$ be continuous and $X$ compact and $K\subset X$ compact. By $s_n(2^{-k},K,T)$ denote the maximal cardinality of any $(n,2^{-k})$ separated subset of $K$. Suppose, we know for ...
103 views

### proof of upper bound on differential entropy of f(X)

I asked a similar question yesterday, but I organized my question here a little and further asked my second question. Suppose $X$ is a continuous random variable with the pdf $f_x$, and $Y=g(X)$. If ...
51 views

### Is there a statistical measure of bitwise entropy?

(Somewhat inspired by this website, particularly Section III. Also, I might be using a different definition of entropy than usual; what I am using is closest to the physics definition (the one I ...
50 views

### How to calculate the Shannon Entropy for a block length of a word

I have a binary sequence of length N as $10110110111...$ I want to segment the above series into equal blocks of a window of length $L$. One way of determining the block length is using the ...
13 views

### pattern-sensible entropy measure

I have some binary images (meaning each pixel can be 0 or 1), I want to find a pattern-sensible entropy measure, which means for example that a chessboard should have a very low entropy value (almost ...
11 views

### Can the entropy of a system of multiple stochastic processes be defined?

I'm trying to measure the entropy of BINGO cards. (waiting for laughter to die down.) Normally one would think of this as 75 choose 24, but in reality it is 15 choose 5, four times, and 15 choose 4, ...
5 views

### Calculating the entropy of original set E(S)

I am currently revising for an exam and I am struggling with this question Using a decision tree algorithm with information gain splitting, which of the two attributes diarrhea or fever is a ...
11 views

### Compare two distributions with varying focus on different regions

I have been trying to find if my problem matches has been discussed in prior research and if any technique exists to solve it. Here's the problem: Given two distributions (pdf) D1 and D2 over a ...
17 views

### Entropy of factor map

Let $A$ and $B$ be two compact metric spaces with $B\subset A$- Moreover, let $T\colon A\to A$ continuous and let $S\colon A\to B$ a continuous surjection with $S\circ T=T\circ S$. Moreover assume ...
97 views

### Random permutations composition

I'm trying to prove a theorem that seems very intuitive. However, I seem to be missing a piece of the puzzle. If: $\pi$ is a random permutation ($S_n$), $\pi_1, \pi_2$ - random permutations with ...
13 views

64 views

### Deriving the q-Gaussian PDF

Ok it may sound a bit too simple but I am quite confused here. While studying generalized entropic forms, in my case that of $S_q$ or in another words the Tsallis Entropy, I reach a point where I have ...
21 views

### Calculating mutual information for a dataset

I have a dataset of individual text documents $D = {d_0, d_1, ..., d_n}$ and a corpus of keywords $K = {k_0, k_1, ..., k_m}$ in the documents. There are zero or more keywords in each text document. I ...
48 views