Tag Info

Hot answers tagged

3

Perhaps the following fact is what you are after: Not every map $\mathbb R^n \rightarrow \mathbb R^n$ is the gradient of a function $\mathbb R^n\rightarrow\mathbb R$. That is, even though the gradient "creates dimensions", as you put it, it is also constrained in its form, which sort of cancels out the new dimensions. For instance, in two dimensions, ...


3

Differential Entropy can actually be negative, which is one of it's drawbacks. It just so happens that on $[0,1]$ all continuous distribution entropies are negative except for the uniform distribution. Let $h(x)$ be any continuous distribution on $[0,1]$ and $u(x)=1$ be the uniform distribution. Here's the proof using KL divergence notation:: $$0\leq ...


2

It is neither convex nor concave. You can work it out using Bernoulli random variables. Not convex: 1) Let $X$ and $Y$ be i.i.d. Bernoulli with $Pr[X=1]=1/2$. Then $I(X,Y)=0$. 2) Let $A=B=1$ (constants). Then $I(A,B)=0$. 3) Let $(W,Z) = \left\{\begin{array}{ll} (X,Y) & \mbox{with prob 1/2} \\ (A,B) & \mbox{with prob 1/2} ...


2

The differential entropy $h(x)$ is not a true generalization of the (discrete, true) entropy $H(X)$, only some of the properties of the later apply to the former. In particular, the property that $H(X)\ge 0$ , with $H(X)=0$ meaning "zero uncertainty" (or full knowledge), does not apply to $h(x)$. The differential entropy can be negative, and $h(x)=0$ has no ...


2

In this sense, memorylesssness of $\left(X_n\right)_{n\geqslant 1}$ implies its stationarity and ergodicity. Ergodicity follows from Kolmogorov's $0$-$1$ law and stationarity by using the fact that the law of $(X_1,\dots,X_n)$ is the product of the common law of $X_n$'s. The converse is not true: if $\left(\varepsilon_i\right)_{i\in\mathbb Z}$ is an ...


1

Yes, the differential entropy disregards resolution (quantization). A continuous random variable can't be represented exactly with finite number of bits. By introducing an approximate representation through quantization you can relate to classical entropy. For example if you quantize uniformly with intervals of lenght $\Delta=2^{-n}$ you get a discrete ...


1

The dimensions are already there. It's $\mathbb{R}^n$ after all. The fact that your function is mapping it onto $\mathbb{R}$ doesn't change the fact that there's plenty of information from every one of the $n$ directions. In fact, that's exactly what the gradient is revealing to you. It's not creating any dimensions nor is it creating any more information; ...


1

Well, let's review a few helpful definitions that will clarify how to calculate the Kullback-Leibler divergence here. By definition the summation of the parameters of the mutlinomial distribution is 1; i.e., $$\sum_{m=1}^k\theta_m=1$$, where $\theta_m$ is the probability of the $m^{th}$ outcome occuring. The probability mass function (PMF) of the ...


1

In Shannon Entropy, MUST the probability be based solely on the sequence itself or can the probabilities be predetermined Rather on the contrary (if I understand you right): the probabilities must be predetermined. More precisely: the Shannon entropy is defined in terms of a probabilistic model, it assumes that the probabilities are known. Hence, it ...


1

Here's a reasonable notion. Let $(X,p_X)$ and $(Y,p_Y)$ be finite probability spaces; we can assume $|X|=|Y|$ by adding elements of probability zero to either space. Then set $$D(p_X,p_Y)=\min_{\pi:X\to Y} \sum_{x\in X} |p_X(x)-p_Y(\pi(x))|$$ where the min is taken over all bijections from $X$ to $Y$. (You need to check that this is well-defined -- if we ...


1

\begin{align*} \sum_x \int \mu(x,y) \log \mu(y \mid x) dy &= \sum_x \int \mu(x,y) ( \log \mu(x, y) -\log p(x) ) dy \\ &= \sum_x \int \mu(x,y) \log \mu(x, y) dy - \sum_x \int \mu(x,y) \log p(x) dy \\ &= \sum_x \int \mu(x,y) \log \mu(x, y) dy - \sum_x \log p(x) \int \mu(x,y) dy \\ &= \sum_x \int \mu(x,y) \log \mu(x, y) dy - \sum_x p(x)\log p(x) ...



Only top voted, non community-wiki answers of a minimum length are eligible