New answers tagged entropy
0
This can be solved because there is 2 unknown var and 2 equations. It is exponantial but it can be solved.
I tried to use octave (http://www.gnu.org/software/octave/)
octave -q filename.math
and filename.math seems like;
function y = f (x)
y = zeros (2, 1);
y(1) = ((0.1*exp(-0.1*x(1)-0.1^2*x(2))) +(0.3*exp(-0.3*x(1)-0.3^2*x(2))) ...
1
A partial answer for further reference:
In short, use the integral formulation of the entropy and pretend that the discrete distribution is sampling a continuous one.
Thus, create a continuous distribution $p(x)$ whose integral is approximated by the Riemann sum of the $p_i$'s:
$$\int_0^1 p(x)dx \sim \sum_i p_i\cdot \frac{1}{N} = 1$$
This means that the ...
2
Use that $\log ab=\log a+\log b$ and $\log_b a=\dfrac{\log a}{\log b}$. Here $\log$ is the natural logarithm. Remember you want to show there exists $C$ such that your expression in absolute value is $$\leq C\frac{\log n}n$$
0
If all $16$ possibilities has an equal probability, that entropy must $\ln 16\approx 2.77$
its; $H\sim \max= \sum[ \frac1n \ln(\frac1n) ]$. Adding new information with "equal" probability is not add extra information into entropy.
1
I'm not sure exactly what you're trying to ask here. You'll probably be able to find this answer in a textbook somewhere. If you were asking something else it would be helpful if you gave a little more detail in your question.
Suppose $p_i< 2^{-k}$. If I toss a coin $k$ times I have $2^k$ possible events each of probability $2^{-k}$. So I cannot ...
1
Here is an elaboration of what Qiaochu has indicated in his answer. Let $R$ be a complete topological commutative ring, which is also a $\mathbb{Q}$-algebra, and whose topology is induced by some ideal $I$ (i.e. the powers $I^k$ are a fundamental system of open neighborhoods of $0$). Then, for every $a \in I$, the sequence defined by $s_n = \sum_{k=1}^{n} ...
2
You can define the logarithm on Banach algebras using the Taylor series whenever it converges, and more generally in any topological ring where the Taylor series makes sense and converges. This is already useful for e.g. finite-dimensional real matrices. But IIRC the correct statement is that $\log(ab) = \log(a) + \log(b)$ if everything is defined and $a, b$ ...
2
By subtracting $\bmod 2$, the question is equivalent to predicting the next value of the sequence. Under the assumption that the sequence is computable, you want a sequence with high Kolmogorov complexity; the higher the Kolmogorov complexity of a sequence, the harder it is for someone to even describe it, let alone guess it. The sequence 1111... (which ...
2
Either can be larger, or they can be equal. http://en.wikipedia.org/wiki/Interaction_information has some informative examples.
0
I guess $A$ and $B$ are discrete random variables. Since $C$ is a function of $A$ and $B$ one has $H(C \mid A) \leq H(A,B \mid A)$ (because this is even true that $H(C \mid A=a) \leq H(A,B \mid A=a)$ for every $a$). Moreover $H(A,B \mid A) = H(B \mid A)$ because it is easy to check that $H(A,B \mid A=a) = H(B \mid A=a)$ for every $a$. Finally $H(C \mid A) ...
Top 50 recent answers are included
