# Tagged Questions

23 views

### Equality of Information Gain and Mutual Information

I am curious about definition of information gain and mutual information in the context of feature selection. If looks like two these measures define exactly the same thing, however I didn't find ...
59 views

### Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
19 views

### Orthogonality of parameter space

Folks, I have a basic information theory question. I am fitting a highly parameterized model to some data. In general: $$y = \sum\limits_{i=1}^{13} \alpha_i X_i$$ Currently I use gradient descent ...
125 views

### KL divergence of multinomial distribution

Consider $q(x)$ be a Multinomial distribution over $\{1, \ldots, k\}$ with parameters $\{\theta_1,\ldots, \theta_k\}$. And p(x) over $\{1,\ldots, k\}$ with distribution $p(x)=\frac{1}{k}$. Then what ...
60 views

### Amount of information a hidden state can convey (HMM)

In this paper (Products of Hidden Markov Models, http://www.cs.toronto.edu/~hinton/absps/aistats_2001.pdf), the authors say that: The hidden state of a single HMM can only convey log K bits of ...
40 views

### How do I measure the similarity of two bivariate time series?

Suppose I have two bivariate time series: $$ts1 = [<a_1, b_1>, <a_2, b_2>, \cdots, <a_N, b_N>]$$ $$ts2 = [<c_1, d_2>, <c_2, d_2>, \cdots, <c_N, d_N>]$$ Which ...
69 views

### Given $\forall x \in \mathbb{R} \: h(p^t(x))=th(p(x))$, how to get $h(p(x)) \propto \ln p(x)$?

The whole question is in the title. $p(x)$ is a probability distribution, and $h$ is continuous and monotonic in $p(x)$. The purpose is to motivate that the "degree of surpise", or the "amount of ...
630 views

### What is the relationship between the Boltzmann distribution and information theory?

I'm reading a paper on Boltzmann machines (a type of neural network in Machine Learning), and it mentions that "The Boltzmann distribution has some beautiful mathematical properties and it is ...
I'm looking to paper "A Kullback-Leibler Divergence Based Kernel for SVM Classification in Multimedia Applications". Author suggest to use kernel function for two distributions $p$ and $q$: \$k(p,q)= ...