1
$\begingroup$

Suppose $X$ is discrete and $Z,W$ are discrete or continuous, I am wondering if it is always the case (or at least non-trivially) that

$$ P(X=x\mid Z=z) \geq P(X=x\mid Z=z,W=w) $$

for all $x\in X$, $w\in W$, and $z\in Z$?

It appears to make intuitive sense to me because it appears the probability of $X$ given $Z$ and $W$ should be "subsetting" off the probability of $X$ given $Z$. That is, if there is some chance of $X$ given $Z$, then $X$ given $Z$ and $W$ is subdividing the occurrence of $X$ given $Z$ into many more chunks according to $W$, and thus the probability of a chunk occurring should be less than the whole.

In other words, it seems to hold in the finite-sampling perspective, where the above are empirical proportions. However, I am unable to prove this generally. Is there a general result behind this?

$\endgroup$
  • $\begingroup$ $\mathbb{P}(X)$ is meaningless when $X$ is a random variable. E.g., if $X$ is the result of a die roll (a number from 1 to 6), $\mathbb{P}(X)$ is "the probability of the result of a die roll" (meaningless). Is $X$ an event? If so, what does it mean for an event to be discrete? $\endgroup$ – parsiad May 14 at 4:19
  • $\begingroup$ Are you implying I should fix $X=x$. I have changed the above. $\endgroup$ – user321627 May 14 at 4:21
  • $\begingroup$ This seems like the basic idea of conditional probability (en.wikipedia.org/wiki/Conditional_probability) . So in the post you did not mention whether $X, Y, Z$ are independent or not. In the case where the variables are independent, then $Z$ and $W$ contribute no information about $X$, no matter whether any of the variables are discrete or continuous. Perhaps you can clarify the dependency structure you are assuming in the post itself. $\endgroup$ – krishnab May 14 at 4:26
  • $\begingroup$ I am assuming that there is no independence existing either marginally nor conditionally among any of the variables. I was hoping for a general result. $\endgroup$ – user321627 May 14 at 4:28
2
$\begingroup$

Certainly not true in general (in fact in 'most' cases). For example suppose $W=X$ and $X$ is independent of $Z$. If you take $w=x$ then the inequality becomes $P(X=x)^{2} \geq P(X=x)$ so it is false if $0<P(X=x) <1$.

$\endgroup$
1
$\begingroup$

The claimed relation is not true in general. As a counterexample, suppose $X$ and $W$ are independent and Bernoulli distributed random variables, with each of them taking the values $0$ and $1$ with equal probability. Further, suppose that \begin{equation} Z=X+W\quad \text{mod }2. \end{equation} Then, it is easy to see that $P(X=0|Z=0,W=0)=1$, whereas $P(X=0|Z=0)=\frac{1}{2}$, and therefore we have $P(X=0|Z=0)\ngeq P(X=0|Z=0,W=0)$.

It seems like you want to capture the fact that "given information about $Z$ and $W$, there is less uncertainty about the value that the random variable $X$ takes, than when given information only about $Z$" (this is true in the example I have mentioned above). This notion of uncertainty reducing upon conditioning more and more random variables is captured very nicely in the subject of information theory through a quantity known as Entropy. Please have a look at this quantity by visiting the link if you have not encountered it before.

So, in the language of information theory, it is true that \begin{equation} H(X|Z)\geq H(X|Z,W), \end{equation} where $H(X|Z)$ and $H(X|Z,W)$ denote the conditional entropy of $X$ given $Z$ and the conditional entropy of $X$ given $Z$ and $W$. In fact, the above relation between the conditional entropies is a consequence of a more general property: the difference between $H(X|Z)$ and $H(X|Z,W)$ is a quantity in information theory known as conditional mutual information of $X$ and $W$ given $Z$, which is denoted by $I(X;W|Z)$, and it is a known fact in information theory that conditional mutual information is always nonnegative.

Hope this clarifies your doubt.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.