New answers tagged

0

Now, is it possible that for some N,K the output of the PRNG function is going to be X? The answer depends on how you understand the question. Is it possible that the output of some PRNG function (with "small" parameters $N,K$) happens to be my target string (my movie) of "big" length $X$? Yes, it's possible. (But the probability is extremely low). ...


0

The statement is not clear, because you don't specify how $Z$ is (jointly) related with the other variables. Assuming you mean that the conditional are the same ($P(X|Y,Z)=P(X|Y',Z')$) : $$H(X|Y,Z)=-\sum_{y,z} p(y,z) \sum_{x} p(x|y,z)\log p(x|y,z) $$ This implies that for both scenarios the inner sum (the entropies conditioned on particular values: ...


0

The question is phrased as whether $$ \sum_{x\in\mathcal X, y\in\mathcal Y}p(x,y)\log\,p(x) = \sum_{x \in \mathcal X} p(x)\log\,p(x) \quad \text{?} $$ I disapprove of using the same symbol, $p$, for two different functions. If one instead writes $p_X(x)$ with capital $X$ and lower-case $x$ in the appropriate places, one can then understand such things as ...


0

Simply because $$ \sum_{y\in\mathcal Y}p(x,y)= p(x). $$ Really, you had done all difficult work already.


0

Information geometry is introduced in Amari's book. A concise treatment of the relation between information theory and statistics can be found in "Information Theory and Statistics : A Tutorial by Csiszar and Shields. Particularly section 3 and 4 are dedicated to information geometry. They are credible researchers on this topic. There is this chapter of ...


0

I found one approach, although we get a $1/(np)$ term rather than $1/n$. According to [1], for any integer-valued random variable $X$ (including a Binomial) with variance $V$, we have \begin{align} H(X) &\leq \frac{1}{2} \log_2 \left[ 2\pi e \left(V + \frac{1}{12}\right) \right] . \end{align} This is already very nice/useful, and probably often quite ...


1

Take your geometric random variable distributed as $p_k=\mathbb P (X=k)=p(1-p)^{k}$ for $k\in\{0,1,2,\dots\}$. Note that your entropy is: $$ H(X)=-\log p-\frac{1-p}{p}\log(1-p). $$ And consider another random variable $Y$ distributed as $q_k=\mathbb P (Y=k)$ such that: $$ \frac{1-p}{p}=\mathbb E(X)=\mathbb E(Y)=\sum_{i=0}^\infty i q_i. $$ We use following ...


0

Not a solution to your question, but just to let you know that for a geometric random variable $X$, $$P(X=k)=p(1-p)^{k-1}$$ From the definition of expectation or by looking at the moment generating function, you should have that $$E(X)=\dfrac{1}{p}$$ The entropy $H(X)$ is then given by $$H(X) = \sum_{k=1}^{\infty} -(1-p)^{k-1}p \cdot ...


1

$\log 1/x_i$ is sometimes known as the 'surprise' (e.g. in units of bits) of drawing the symbol $x_i$, and $\log 1/X$, being a random variable, has all the operational meanings that come with any random variable, namely, entropy is the average 'surprise'; similarly, higher moments are simply higher moments of the surprise measure of $X$. There is indeed a ...


2

$$bb(x)=log_2(x)$$ is the "binary base" , the logarithm to base $2$.


1

As you say, $$\theta_{t}\text{d}B_{t} = \theta_{t}\text{d}W_{t}+\theta_{t}^2\text{d}t\ ,$$ where $W$ is a $Q$-Brownian motion. Hence $$\log\frac{\text{d}Q}{\text{d}P} = \int \theta_{t}\text{d}W_{t} + \frac{1}{2} \int \theta_{t}^2\text{d}t$$ and the $P$-expectation becomes $$H(Q;P)=\mathbb{E}_{Q}\left(\int ...



Top 50 recent answers are included