Confusion between probability distribution function and probability density function

Question

As far as I am concerned, probability distribution function is for discrete random variables while probability density function is for continuous random variables. To find the probability value of continuous random variable, we have to take the total area under the function which differ from discrete random variable, where we can take the probability value directly from the function.

But here's my confusion,

Let's say Z=X+Y. X is discrete and can take in value of -1 and 1. Y is Gaussian random variable. So,Z is Gaussian random variable with probability density function of p(Z)=p(Z|X=1)+p(Z|X=-1).

However when we try to find p(X=1|Z), it is equals to [p(Z|X=1)p(X=1)]/p(Z). My questions, how can we time p(Z|X=1) and p(X=1) since former is probability density function while the latter is probability distribution function? What's more, p(Z) is also a probability density function. In the end, what is p(X=1|Z), a probability density or distribution function?

Answer 1

the real complication here is the joint probability distribution of $X$ and $Y$. In introductory courses, one learns about the joint pdf of two continuous random variables, or the joint pmf of two discrete random variables. The mixed case you mention, where one variable is continuous and the other is discrete is not commonly mentioned in introductory texts - although some manage to sneak it in without calling attention to just what they are dealing with.

A common example is this: suppose $X$ has the binomial distribution bin($n,u$), where $u$ is the success probability. Each day a new $X$ is observed. However, the value of $u$ changes from day to day in a random way: it is selected each time at random uniformly from the interval (0,1). Because the success probability $u$ is also random, we should denote this random quantity [before it is selected] by $U$. [In Bayesian contexts, this uniform distribution for $U$ is referred to as a 'prior pdf' for $U$. however, unlike the bayesian context, which focuses on $U$, our focus here is on $X$.]

so we have here a mixed pair of random variables $(X,U)$, where one is discrete and one is continuous [as in the OP's question.] here the [mixed] joint probability distribution of $(X,U)$ is specified by two distributions: the [marginal] pdf of $U$: $f_U(u) = 1$ for $0 < u < 1$ [and is $0$ elsewhere] - and what is really the conditional distribution of $X$ given $U$, for which the conditional pmf is

$$f_{X|U}(x|u) = P(X=x|U=u) = P(bin(n,u) = x) = {n\choose x} u^x(1-u)^{n-x},\quad x = 0,1,\cdots, n.$$

it turns out that the joint [mixed] pmf-pdf of $(X,U)$ can be written as the product of the marginal pdf of $U$ and the conditional pmf of $X|U$, analogous to what one does in the case where both variables are discrete, or both are continuous. [the justification for this involves what kahen alludes to in his answer - but it goes beyond the scope of this reply.]

so we can write the mixed joint pmf-pdf of $(X,U)$ as

$$ f_{X,U}(x,u) = f_U(u)f_{X|U}(x|u) = {n\choose x} u^x(1-u)^{n-x},\quad x = 0,1,\cdots, n,\quad 0<u<1.$$

[here $f_{X,U}(x,u)$ is understood to vanish for other values of $(x,u)$.]

this mixed pmf-pdf can be treated similarly to the way one manipulates a joint pmf or pdf [to get marginal and conditional distributions, for example] - except that one integrates out $u$ to get the marginal pmf of $X$ and sums out $x$ to get the marginal pdf of $U$.

so for example:

$$f_X(x) = \int_0^1 f_{X,U}(x,u)du =\int_0^1 {n\choose x} u^x(1-u)^{n-x}du = \frac{1}{n+1},\quad x = 0,1,\dots,n.$$

thus $X$ is marginally uniformly distributed on the set of outcomes $\{ 0,1,\dots, n\} $. [this means that over a large number of days, all of the possible values of $X$ (from $0$ to $n$) will turn up about equally often. given the way $U$ varies from day to day, this seems plausible.]

$\large \mathbf [$ the evaluation of the integral is straightforward on noting that it is a beta function integral, so that

$$\int_0^1 u^x(1-u)^{n-x}du = Beta(x+1, n-x+1) = \frac{x!(n-x)!}{(n+1)!}. {\mathbf ]}$$

one can similarly get the conditional pdf of $U|X$ as

$$f_{U|X}(u|x) = \frac{f_{X,U}(x,u)}{f_X(x)} = (n+1){n\choose x}u^x(1-u)^{n-x},\quad 0<u<1.$$

this means that looking only at the days when $X=x$, conditionally, $U$ has a beta distribution with parameters $(x+1, n-x+1)$. this result is what interests bayesians in their contexts - altho their scenario would usually involve considering the value of $X$ only for day 1, and their interpretation of $f_{U|X}(u|x)$ isn different from that given here.

the bottom line is that in working with a mixed pmf-pdf, each variable retains its discrete or continuous nature whether one considers marginal, joint or conditional distributions, and the joint mixed pmf-pdf can be manipulated as in the pure [= unmixed] cases, remembering to sum or integrate as is appropriate.

this reply does not address the specific problem posed in the OP, but it can be handled by the same methods. [only the imputs are a bit different: there one starts with [the joint pmf-pdf of] $(X,Y)$ and proceeds to $(X,Z)$. starting from the mixed pmf-pdf of $(X,Y)$, it is straightforward to get the conditional pmf of $Z|X$ and then the joint mixed pmf-pdf of $(X,Z)$.]

Answer 2

You have assumed $X$ to be discrete, so $P(X=1|Z)$ had better be discrete or something really weird is going on. Just because you know $Z$ doesn't still change that $X$ still "lives on" a countable set.

It will all make a lot more sense after you've had measure theory and you see that it's all really the same thing:

In the continuous case, the probability that you get an outcome in $B \subset \mathbb{R}$ is $\int_B p(x) \; \mathrm{d}m(x)$, where $p$ is your density and $m$ is the Lebesgue measure. "Of course" this can be rewritten into $\mu(B)$ for some probability measure $\mu$.

In the discrete case, the probability that you get an element in $A \subset \mathbb{N}$ is exactly $\sum_{a \in \mathbb{N}} p(a)$, but this can be rewritten as an integral with respect to the counting measure $c$: $\int_\mathbb{N} p(a) \; \mathrm{d}c(a)$. And this too can be rewritten into a probability measure: $\nu(A)$, where $\nu = \sum_{a \in \mathbb{N}} p(a) \delta_a$, $p$ being your distribution and $\delta_i$ the Dirac measure at $i$.

Everything is "just" integrals with respect to some measure. And finally a word of warning: one shouldn't forget that the statement: "Probability theory is just measure theory in the special case of total measure one" is a gross oversimplification.

Answer 3

Let $X$ be random variable over $\mathbb{Z}$ with probability weights $p_k = Pr[X=k]$. Many discrete scenario can be expressed in these terms.

The probability distribution function is by virtue of definition $F(x) = Pr[X \leq x] = \sum\limits_{k=-\infty}^{\lfloor x \rfloor}Pr[X=k] = \sum\limits_{k=-\infty}^{\lfloor x \rfloor} p_k$.

A density function corresponds to the weights $p_k$ while the definition of distribution function remains the same in the continuous case, only that $Pr[X \leq x]$ has to be calculated by an integral instead of a sum. See kahen's answer for details.

So no, you should not exchange distribution function and density. They do different things.