0
$\begingroup$

Say I have two (somewhat related) random variables as follows:

$S \sim \mathcal{N}(s \mid \mu^2, \sigma^2) \implies \mathcal{P}(s) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(s-\mu)^2}{2\sigma^2}}$ $\begin{align*} P \sim \mathcal{N}(p\mid s,\beta^2) &\implies \mathcal{P}(p) = \frac{1}{\sqrt{2\pi\beta^2}} e^{-\frac{(p-s)^2}{2\beta^2}}\\ &\implies \mathcal{P}(p) = \frac{1}{\sqrt{2\pi(\sigma^2+\beta^2)}} e^{-\frac{(p-\mu)^2}{2 (\sigma^2 +\beta^2)}} \end{align*}$

and I want to see what $\mathcal P(s, p)$ looks like.

From the standard definition of a joint distribution:

https://en.wikipedia.org/wiki/Joint_probability_distribution#Continuous_case

I can surmise that:

$\mathcal P(s, p) = \mathcal P(p\mid s)\mathcal P(s)$

which is the product of two Gaussian forms. I am further given to know (in Wikipedia and other references) that the results $\mathcal P(s, p)$ is a PDF and hence:

$$\int_{-\infty}^\infty \int_{-\infty}^\infty \mathcal P(s,p) \, ds \, dp = 1$$

But, if I multiply the two Gaussians together and simplify, I end up with a result that is, not a PDF! Much rather it is a scaled version of a PDF. And Bromily basically does all the same algebra and simplifications and summarises them neatly with the same conclusion here:

http://www.tina-vision.net/docs/memos/2003-003.pdf

Bromily concludes that: "the product of two Gaussian PDFs $f(x)$ and $g(x)$ is a scaled Gaussian PDF and the scaling factor $S$ is itself a Gaussian PDF on both $\mu_f$ and $\mu_g$ and with standard deviation $\sqrt{\sigma_f^2+\sigma_g^2}$."

The key observation being that the integral is not 1 it is not a PDF, there is this scaling factor that emerges from the multiplication.

So who is right? Or better said given I trust both are right, what error am I making in equating Bromily's result with the definition of a joint PDF? Where am I in error?

If it's any help I have my whole conundrum summarised here:

https://drive.google.com/open?id=1f2ZevrUoPWmQegmiXYBxigkXDqoCIwFH

with the complete derivations bar the last step where I defer to Bromily as it got messy fast.

$\endgroup$
1
$\begingroup$

Your error is in applying Bromiley's result to a situation where $\mathcal P(p\mid s)$ is a function of both $p$ and $s$. In other words, $p$ is not constant when we assert that $\mathcal P(s, p) = \mathcal P(p\mid s)\mathcal P(s)$ has a joint Gaussian distribution; instead the joint pdf $\mathcal P(s, p)$ is jointly Gaussian as a function of two variables.

If you regard $\mathcal P(p\mid s)$ as a function of $s$ only, with $p$ constant, then yes, the product of $\mathcal P(p\mid s)$ with $\mathcal P(s)$, when regarded as a function of $s$ alone, is not a density, as Bromiley shows. But nobody is claiming that $\mathcal P(p\mid s)\mathcal P(s)$ is a density when regarded as a function of $s$ only.

Bromiley remarks that the scaling constant $\gamma$ does involve both $\mu_f$ and $\mu_g$. This means that $p$ is hiding in the scaling constant, since your summarization document renames $p$ as $\mu_g$. So the fact that $\mathcal P(p\mid s)\mathcal P(s)$ is not a density when viewed as a function of $s$ alone does not contradict the fact that it is a density in two variables $s$ and $p$.

(To obtain the marginal density of $P$, you must integrate out $s$ via ${\mathcal P}(p) = \int_s \mathcal P(p\mid s)\mathcal P(s)\,ds$. The result of this integration will be a density. The marginal density of $S$ is ${\mathcal P}(s)$ as given.)

$\endgroup$
  • $\begingroup$ Thanks so much for the quick and informative pointer I will explore that integral to see what I can get, but in the interim for clarification I'll address "nobody is claiming", because I find the Wikipedia article cited says clearly "these are probability distributions" referring to the product. If it means only the two operands (things being multiplied) not the product itself it could be clarified, and being Wikipedia it can be easily. Which I'm happy to do if and when I'm truly able to say I understand this (i.e tried that integral and see if I can get anywhere). $\endgroup$ – Bernd Wechner May 24 '18 at 6:53
  • $\begingroup$ @BerndWechner I agree the Wikipedia article is rather vague about the assertion "since these are probability distributions". It refers to the fact that $f_{X,Y}(x,y)$ is a joint pdf in $x$ and $y$. $\endgroup$ – grand_chat May 24 '18 at 7:03
  • $\begingroup$ From what I understand then, the produce is a joint PDF, meaning a PDF in two variables, so the integral over both is 1 but the integral over only 1 is in fact not resolved (still written in terms of the other). Either way I'd love to take a crack at that double integral, on the back of Bromily's work, ready to accept that I fail (that it's not readily integrable) or that it proves doable. Haven't found such a demonstration yet in elegant literature like Bromily's (that walks through a proof). $\endgroup$ – Bernd Wechner May 24 '18 at 7:44
  • $\begingroup$ Also thanks for pointing out the errors in my mapping of $p$ and $\mu_g$, fixed my PDF so it reflects the appropriate problem substitutions from the generalisation now. $\endgroup$ – Bernd Wechner May 24 '18 at 7:51
  • $\begingroup$ Done, integrated it and got 1. Thanks! I will summarise the proof tomorrow I hope and post. $\endgroup$ – Bernd Wechner May 24 '18 at 13:14
0
$\begingroup$

The source of confusion has now been identified and resolved as follows. It centered on Bromily's conclusion that:

"the product of two Gaussian PDFs $f(x)$ and $g(x)$ is a scaled Gaussian PDF and the scaling factor S is itself a Gaussian PDF on both $μ_f$ and $μ_g$ and with standard deviation $\sqrt{σ^2_f+σ^2_g}$."

and misinterpretation of it. Essentially, he presents the product in a new form which is itself a product of two Guassians cast in new terms. The both integrate over the range of $-\infty$ to $\infty$ to 1.

The key observation is that the scaling factor is not a constant, but itself a function. In the specific TrueSkill case I was working on this translates to:

$\mathcal P(s,p) = \mathcal P(p \mid s)\mathcal P(s)= \gamma \mathcal N (s \mid \mu_{sp}, \sigma_{sp}^2)$

Where $\gamma$ is the scaling factor and $\mathcal N (s \mid \mu_{sp}, \sigma_{sp}^2)$ is the Gaussian PDF and:

$\mu_{sp} = \frac { \beta^2\mu + \sigma^2 p } {\sigma^2 + \beta^2}$

$\sigma_{sp}^2 = \frac {\sigma^2 \beta^2} {\sigma^2 + \beta^2}$

$\gamma = \frac {1} { \sqrt{2\pi(\sigma^2+\beta^2) }} e^{ -\frac{(\mu-p)^2}{2(\sigma^2+\beta^2)} }$

And so we see that the scaling factor $\gamma$ is not a constant but rather a function of $p$. And integrating it with respect to p over the range of $-\infty$ to $\infty$ also yields 1 (as it is in the general form of a Normal PDF).

To wit, no conflict exists, between the general claim and application of Bromiley's result for the Gaussian multiplication and confusion arose form th premise that the scaling factor was a constant.

Important also in understanding the breakdown I found is to note that a joint PDF (with two random variables) describes a surface (is a function of two independent variables) and must be integrated twice to yield the volume beneath the surface which by definition is 1. A standard PDF by comparison is a an ordinary function (a line) and we need integrate it only once with respect to its independent variable to obtain the area beneath the line, which must, over the range of possible values, be 1, for it to be a PDF)

When we integrate Bromiley's result twice to remove both unknowns, we also see a result of 1, and all is good. Integrating it only once leaves us with a function (another PDF) not with a result. And the scaling factor is in fact a function (and to my mind poorly named as a scaling factor for that reason but that is an aside).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.