# Tagged Questions

21 views

### Self-information, one event half as likely than another event conveys twice the amount of information?

I was reading the following: "If one event is half as likely as another, then learning about the former event shouldconvey twice as much information as the latter" I know it should be easy to ...
44 views

### How is this paper using probability notation?

I am trying to understand this paper about documents and sentences. At the end of page three, they say: Let g(wi, wj ) be the distance between two events (1 if in the same sentence, 2 in neighboring, ...
23 views

### Mathematics branch concerned with availability of information

Is there a branch of mathematics that study about availability of information? For example, if I want to search for something on the internet, is there a branch of mathematics that can predict how ...
39 views

### Upper bound on the entropy of a sum two random variables

Let $X$ be a random variable such that $|X| \leq A$ almost surely, for some $A > 0$. Let $Z$ be independent of $X$ such that $Z \sim {\cal N}(0, N)$. My question is: How large can the entropy ...
46 views

137 views

### Similarity between two probability distribution

I am not sure how to put the question. I am not even sure if this question makes sense at all. I know that the similarity of two discrete (or continuous) distributions can be quantified by ...
34 views

### A question on Markov chain

Suppose for two random variables $X$ and $Y$ we have $X\perp\!\!\!\perp Y$ and also assume that three random variables $X$, $Y$ and $Z$ form the following Markov chain: $X\to Z\to Y$. Do these two ...
40 views

### Problem with calculating probability of symbols

I've a $100 \times 100$ binary matrix it`s constructed with this probability table : i want to apply extended Huffman on this matrix my idea is to compress each column individually . - so starting ...
33 views

### Calculating Entropy of Dependent Random Variables

So basically I'm trying to answer the following exam problem: I'm half struggling on H(Z | X) and H(X | Z) and mainly just need confirmation. I know that H(Z | X) = -SUM P(Z|X)P(X)logP(Z|X) ...
81 views

### Bounding second moment of entropy

Entropy is defined as $E(-\log(P(x))$. We know it is bounded by $\log(r)$ when $r$ is the size of alphabet. Defining the second moment as $E(\log^2(P(x))$, how to show it is bounded?
61 views

### What's the name of the quantity $\mathbb{P}(A\cap B)/(\mathbb{P}(A)\mathbb{P}(B))\;$?

In a physics book, I've come across the quantity $$\frac{\def\P{\mathbb{P}}\P(A\cap B)}{\P(A)\P(B)}\,,$$ where $A$ and $B$ are events. The author calls this quantity the correlation of $A$ and ...
59 views

### Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
53 views

### Shannon Entropy Minimization

The Shannon Entropy for an observation is given by $-x \log_2(x)$. Why is the maximum entropy achieved at $x = \frac{1}{e}$, and not at $x = 0$? Could someone provide a logical explanation that ...
84 views

40 views

### Upper bound of mutual information in a Markov chain

Consider binary random variables $X$ and $V$ with marginal distributions $p$ and $\pi$ respectively and also the conditional distribution $p(X=x\mid V=v)=q(x\mid v)$, where $x\in\{-b,b\}$ and ...
60 views

### Negative exponential/ exponential power distribution between 0 .0 and 1.0?

Note: I'm not very familiar with distribution and higher level math Heyho, I'm currently looking for a way to generate random values between 0.0 and 1.0 with an exponential power or negative ...
41 views

### Prove that communication protocol complexity less than $n\epsilon$

Alice and Bob get as an input words $x$ and $y$, which consist of $0$ and $1$. Length of $x$ is $n$ and length of $y$ is $2n$. They want to know if the word $x$ is subword of word $y$. For example, ...
15 views

### Question about flipping terms in matrix multiplication in proving that $h(N_n(\mu , K))=\frac{1}{2}\log(2 \pi n)^n |K|$

So in my book, it is written: Let $X_1,X_2,...,X_n$ have a multivariate normal distribution with mean $\mu$ and covariance matrix $K$ and $\textbf{X}=(X_1,X_2,...,X_n)$ The above isn't really ...
32 views

### Independent transformation of probability measures

I have a pair of dependent random variable $(\theta, X)$ where $\theta\in K$ for a compact subset $K\subset\mathbb{R}$ and $X\in\mathbb{R}^d$ with marginals $P_{\theta}$ and $P_X$. I want to ...
67 views

### what is relative entropy between to random binary string with length of $L_1$ & $L_2$?

I want calculate relative entropy between two strings of binary such as: $L_1:11000100011101001$ $L_2:00101110110111001$ It is primarily when the lengths of two strings is same and in general when ...
26 views

### $Y$ is a function of $X$: making an inference based on the markovity of $X$

In the information theory book by Cover and Thomas it is written: if $X$ is markov and $Y$ is a function of $X$ then: ...
63 views

### Confused about notation: difference between $\prod_{i=1}^np(x_i)$ and $\prod_{i=1}^np(x)$

In my information theory book by Cover and Thomas, at the beginning of the channel coding theorem, it's written: "Each entry in this matrix" (the matrix of the randomly generated code) "is ...
74 views

### A quick chanllenge: height and weight probability problem

The average height and weight of a group of people is 175cm and 70kg; Find the upper bound of the portion of the people who are over 200cm and over 100kg. I thought about Markov inequality, but I ...
80 views

### Source Words & Huffman Codes

A source $S$ has source words $w_1, w_2, \ldots, w_n$, with probabilities $p_1 \geq p_2 \geq \ldots \geq p_n > 0$. Let $C$ be a binary Huffman code for $S$, and let $l$ be the length of the longest ...
514 views

### Fano's inequality explained intuitively?

I am now reading through a book to understand Fano's inequality, but I remember my professor explaining it in a certain way that made it seem so logical. I will go office hours as soon as possible, ...
152 views

### If $f_\theta=Uniform(\theta,\theta +1)$, a sufficient statistic for $\theta$ is… but why?

If $f_\theta=\mathrm{Uniform}(\theta,\theta +1)$, a sufficient statistic for $\theta$ is $$T(X_1,X_2,\dots,X_n)=(\max\lbrace X_1,X_2,\dots,X_n\rbrace,\min\lbrace X_1,X_2,\dots,X_n\rbrace).$$ Can ...
170 views

### Markov chains for beginners, how to think about them?

So this is what my book states: Random variables $X,Y, and Z$ are said to form a Markov chain in that order denoted $X\rightarrow Y \rightarrow Z$ if and only if: $p(x,y,z)=p(x)p(y|x)p(z|y)$ ...
253 views

### Random process, stochastic process explained intuitively?

So I've read the definitions online and this is what I understood. $X(t)$ is a random process for $t>0$ and we can think of it as being a random variable at any given time $t=t_0$. For example, ...
285 views

### “by definition A and B R.V are independent means that: $p(A∪B)=p(A)+p(B)$ right?” No, absolutely not right.

Can someone please explain why? Isn't $p(a,b)=p(a)*p(b)$ equivalent to $p(A∪B)=p(A)+p(B)$? If not can you please give a counterexample or something? Thanks a lot!
87 views

### I.I.D what does this stand for?

So almost everywhere in the book it's written "random variables are IID", what does this mean? I think it means independent and identically distributed but not sure. So by definition A and B R.V are ...
37 views

### Is $p(X \in A|\frac{Y+Z}{2}) = p(X \in A|Y,Z)?$

Let $X,Y, and \space Z$ be random variables. Let $A$ be a subset of $U$ such that $p(X \in U)=1$ Is $p(X \in A|\frac{Y+Z}{2}) = p(X \in A|Y,Z)?$ Do these two expressions represent the same thing? ...
### Show that entropy $(p1,…,pi,…,pj,…,pm)$, < entropy $(p1,…, (pi+pj)/2 ,…, (pi+pj)/2 ,…,pm)$.
Show that the entropy of the probability distribution, $(p1,...,pi,...,pj,...,pm)$, is less than the entropy of the distribution $(p1,..., (pi+pj)/2 ,..., (pi+pj)/2 ,...,pm)$. I don't understand what ...