suppose that we have a family of probability mass functions ${f_\theta }\left( x \right)$ indexed by $\theta$, and let $x$ be a sample from this distribution. Then from the information theory, we have the following relation:

$I\left( {\theta ,T\left( x \right)} \right) \le I\left( {\theta ,x} \right)$

where ${T\left( x \right)}$ is a function of the samples. The above relation says: by processing the samples, we do not get any new information. Now my question is that, is this claim aways true? For example, consider the following scenario:

Again, assume that we have a family of probability mass functions ${f_\theta }\left( x \right)$. Also, we are given a matrix that shows some correlations among x's dimensions. Now, we process the samples $x$ by this matrix to get new samples as,

$y = Ax$

where matrix $A$ is a similarity matrix indicating some correlations between $x$'s dimensions that we are given. Does $y$ contain new information in my example?

  • 1
    $\begingroup$ The short answer is no, it does not, in a quantitative sense. The question of what that actually means on a practical basis is a much longer discussion that ultimately leads the question of precisely what mutual information measures. In that regard, there are other accounting systems for information content that more clearly identify what is meant by quantitative "information measurement", but still exhibit equivalent data processing inequalities. $\endgroup$ – John Polcari Jul 16 '18 at 13:45
  • $\begingroup$ I should add that this assumes that the original x is the same vector of samples from which you compute y. If you meant "can the vector y provide additional information compared to any one sample in the vector x", then yes, it can (but doesn't necessarily), but only because of any additional information provided by the other samples in the x vector. $\endgroup$ – John Polcari Jul 16 '18 at 14:19
  • $\begingroup$ Thank you very much for your response and explanation. What do you mean by "the original x is the same vector of samples from which you compute y" ? $\endgroup$ – user51780 Jul 16 '18 at 14:55
  • $\begingroup$ At the very start of your question, you mention x being "a sample", which might be taken to mean an individual sample or, alternatively, the vector of samples used to compute y. I initially assumed the latter, because x represents a vector later in your question. It then dawned on me that you might have meant the former, which prompted the additional comment. $\endgroup$ – John Polcari Jul 16 '18 at 15:01
  • $\begingroup$ Sorry for the confusion. In my problem, I compute $y$ for each sample of $x$.Therefore, $y$ is a new random variable. My issue is that is this new random variable provide extra information about $x$. $\endgroup$ – user51780 Jul 16 '18 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.