4
$\begingroup$

One can frequently read, that the product of the densities of two INDEPENDENT random variables is also a density - the joint density of the two variables.

(see for example: http://en.wikipedia.org/wiki/Joint_probability_distribution#Joint_distribution_for_independent_variables)

One can also read, that IN GENERAL the product of two normal pdf is a Gaussian, but not a normal pdf. i.e. one would have to multiply the product with a scaling factor (normalization constant) to get a normal pdf.

(see for example: http://www.tina-vision.net/docs/memos/2003-003.pdf page 3, first paragraph)

My naive interpretation of this would be, that in case of independence, the normalization constant equals 1. But the formula given for the scaling factor in the second source does not seem to support this...

Where is my misunderstanding?

Now, for the multivariate case, i.e. the product of two joint pdf, each one being the joint pdf of a vector of jointly normal variables, but the two vectors being independent of each other, one finds:

"The vectors x1, x2 are statistically independent if their joint distribution is f(x1, x2) = f(x1)f(x2) or, equivalently, if f(x1|x2) = f(x1) and f(x2|x1) = f(x2)."

(I do not have enough reputation points to post more than two links, so I replace the "tt" in http with "**":

h**p://www.le.ac.uk/users/dsgp1/COURSES/THIRDMET/MYLECTURES/5XMULTISTAT.pdf, the above quote can be found on page 3, number 6)

On the other hand, it says in another source: "Suppose f(x) = N (x;1;1) and f(y) = N (x;2;2) are two INDEPENDENT d-dimensional Gaussian densities. Sometimes we want to compute the density which is proportional to the product of the two Gaussian densities, i.e. f(z) = cf(x)f(y), in which c is a proper normalization constant to make f(z) a valid density function."

h**p://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=28A2A856531E5FF9FABEC67397A87B5D?doi=10.1.1.1.2635&rep=rep1&type=pdf" - above quote is the first paragraph in section 7 on page 9 - here slightly modified notation.

These two last sources seem to contradict each other. Is there a reconciling fact I'm missing?

Finally, the previous source given above provides an application in section 8.2 on page 11, where the Bayes theorem is applied to estimate the vector of mean values of a multivariate normal, the Bayes theorem is written there as follows:

f(y|x) = cf(x|y)f(y)

with c being a "normalization constant".

more "traditional" descriptions of the Bayes theorem look more like this:

f(y|x)f(x) = f(x|y)f(y)

(see e.g.: "h**p://en.wikipedia.org/wiki/Bayes%27_theorem#For_random_variables" (slightly rearranged here)

This seems to imply, that the "normalization constant" is actually (always) 1/f(x).

Is this correct?

Best,

JQ

$\endgroup$
1
  • $\begingroup$ May I kindly ask the user who down voted the question to explain why? Obviously there went significant effort into this, references were given, and the question was answered in detail. there may be others who have a similar misunderstanding, for whom to read this post might be helpful! I'm happy to improve the question, if there are any particular issues. But what is the value added by an anonymous down vote, without comment, on a question that cites online sources, got an elaborate answer and additional information in the comments? $\endgroup$
    – user70160
    May 4, 2015 at 2:04

1 Answer 1

8
$\begingroup$

Where is my misunderstanding?

It's the distinction between $f(x)g(y)$ and $f(x)g(x)$. If $f$ and $g$ are densities, then $h(x,y):=f(x)g(y)$ is the joint density of $(X,Y)$ where $X\sim f$ and $Y\sim g$. This is true always, and no additional normalization is required. OTOH, the papers you are looking at are studying $h(x):=f(x)g(x)$, i.e., they are trying to identify the distribution of a single random variable/vector defined by the product of two densities. In general there is no reason to expect that $f(x)g(x)$ integrates to $1$, so you need to normalize. For example, $f(x):=e^{-x}I(x\ge0)$ and $g(x):=2e^{-2x}I(x\ge0)$ are proper univariate densities but $f(x)g(x)$ is not. [But $f(x)g(y)$ is a proper bivariate density.]

On the other hand, it says in another source: "Suppose f(x) = N (x;1;1) and f(y) = N (x;2;2) are two INDEPENDENT d-dimensional Gaussian densities. Sometimes we want to compute the density which is proportional to the product of the two Gaussian densities, i.e. f(z) = cf(x)f(y), in which c is a proper normalization constant to make f(z) a valid density function."

This source is also discussing $f(x)g(x)$ [N.B. the original text reads $p(x)=\alpha p_1(x)p_2(x)$], but seems full of typographical errors. It writes $p(z)$ where it should read $p(x)$. Also, I don't see where the independence is used, unless "independent" has the sense "possibly different".

This seems to imply, that the "normalization constant" is actually (always) 1/f(x).

You are correct! The normalization doesn't matter in the sequel, so it's written $\alpha$ for convenience.

$\endgroup$
10
  • $\begingroup$ Many thanks! You have clarified my misunderstanding. $\endgroup$
    – user70160
    Apr 5, 2015 at 16:14
  • $\begingroup$ So to apply Bayes' theorem, one could define the prior density of a vector with jointly normally distributed elements as f(y), the density of an information set x as f(x), the posterior density (the conditional density of y given that x=X) as f(y|x=X), and the conditional density of the information set given y=Y as f(x|y=Y). Then according to the Bayes theorem, the posterior density is: f(y|x=X)=f(x|y=Y)f(y)/ f(x) and hence the posterior density is a product of normal densities (which is a Gaussian), that is normalized by multiplying with 1/f(x) - and is hence a normal density as well? $\endgroup$
    – user70160
    Apr 5, 2015 at 16:36
  • $\begingroup$ grand_chat - sorry, tried to vote your answer up, but can't, as I don't have 15 recommendations $\endgroup$
    – user70160
    Apr 5, 2015 at 17:03
  • $\begingroup$ Correct. In Bayesian inference $f(y)$ is the prior density of the unknown parameters and $f(x|y)$ is the density of observed data conditional on the parameters. The fact that the posterior density $f(x|y)f(y)/f(x)$ is gaussian/normal when viewed as a function of $y$ relies on two pieces of info: first, $f(y)$ is gaussian, and second, $f(x|y)$ is gaussian when viewed as a function of $y$. (Note $f(x|y)$ is already gaussian as a function of $x$; now we are viewing $f(x|y)$ from the point of view of the parameter $y$.) $\endgroup$
    – grand_chat
    Apr 5, 2015 at 17:32
  • 1
    $\begingroup$ @MrBear Your question is asking about $E(g(X))$ where $g$ happens to be a density. There's no need to normalize in this case. Normalization is necessary if you are defining a new random variable by declaring that the new random variable must have a density proportional to $f(x)g(x)$. That's not happening in your question. As the answerer to your question states, "Whether $u$ has additional meaning is irrelevant." $\endgroup$
    – grand_chat
    Apr 13, 2023 at 17:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .