7
$\begingroup$

Suppose that a signal or observation $s_1$ is drawn from the normal distribution $\mathcal{N}(\mu,\sigma^2)$, where $\sigma^2$ is known but $\mu$ is not. We want to estimate $\mu$ based on $s_1$.

Suppose further we have a normal prior distribution for $\mu$, which is $\mathcal{N}(\mu_0,\sigma_0^2)$.

In this case it is easy to determine the posterior distribution given $s_1$, which is normal (normal is a conjugate prior), and the posterior mean is also easy to determine as $$E[\mu|s_1]=\frac{\mu_0/\sigma_0^2+s_1/\sigma^2}{1/\sigma_0^2+1/\sigma^2}.$$

Now to my question: But what if we cannot observe $s_1$ directly; instead, we only know whether the realization of $s_1$ is above or below a certain threshold $t\in\mathbb{R}$. That is, instead of observing $s_1$, we only observe $\mathbf{1}\{s_1\ge t\}$ ($\mathbf{1}$ is the indicator function).

Since the "evidence" is now an interval rather than a point realization, how to compute the posterior mean $E[\mu|\mathbf{1}\{s_1\ge t\}]$? Is the posterior distribution even normal? I am at a loss here. Any help or references to help would be greatly appreciated.

Edit: I computed the posterior distribution numerically. See the plot below (where the "signal" indicates the realization is above a threshold). The posterior density is clearly not symmetric, hence not normal. So the question remains: Is there a closed form expression for the posterior density, or a somewhat simple expression for the posterior mean? enter image description here

$\endgroup$
2
  • $\begingroup$ Can you tell how do you numerically compute the posterior and for what values of the parameters? $\endgroup$ Aug 9 '18 at 11:11
  • $\begingroup$ I computed it in Matlab, which allows for easy numerical integration. I used the following parameter values for the above figure: prior mean $\mu_0=2$, precision of prior distribution $1/\sigma^2_0= 1$, precision of signal $1/\sigma^2=10$, threshold $t=1$. I chose the relative large differential between prior and signal precision so that the signal gets a large weight, which increases the asymmetry in the posterior. $\endgroup$
    – Nameless
    Aug 9 '18 at 13:40
5
$\begingroup$

Definitions

  1. $X \sim \mathcal{N}(\mu,\sigma^2)$, with $\sigma$ known and $\mu$ the parameter of interest

  2. $\mu_0$ and $\sigma_0$ are hyperparameters describing the prior probability distribution of $\mu$ such that $\mu\sim\mathcal{N}(\mu_0,\sigma_0)$ and $p(\mu|\mu_0,\sigma_0) = \mathcal{N}(\mu_0,\sigma_0)$

  3. $x$ is the observation, $p(\mu|x,\mu_0,\sigma_0)$ is the posterior probability we seek.

Bayesian Inference

Here's the Wikipedia reference for the formulas. $$p(x|\mu_0,\sigma_0)=\int p(x|\mu,\sigma)p(\mu|\mu_0,\sigma_0)d\mu$$ $$p(\mu|x,\mu_0,\sigma_0) = \frac{p(x|\mu,\sigma)p(\mu|\mu_0,\sigma_0)}{p(x|\mu_0,\sigma_0)}$$

Interval evidence

Since we do not have a fixed $x$ but some evidence of the kind $x>t$, we must adjust the probabilities of $x$, $p(x)$ and change them to $f(t)=1-c(t)$ where $c(t)$ is the cumulative distribution function.

The $c(t)$ for a normal function is an appropriately scaled sigmoid function, which in case of a standard normal is $\frac12 +\frac12\text{erf}(\frac{x}{\sqrt{2}})$, where $\text{erf}$ is the error function.

$p(x|...)$ now becomes $p(x\geq t|...) = f(t|...)$.

Calculations

$$f(t|\mu,\sigma) = \frac12 - \frac12\text{erf}\left(\frac{t-\mu}{\sigma\sqrt2}\right)$$ $$p(\mu|t,\mu_0,\sigma_0) = \frac{f(t|\mu,\sigma)p(\mu|\mu_0,\sigma_0)}{\int f(t|\mu,\sigma)p(\mu|\mu_0,\sigma_0)d\mu}=\frac{(\frac12-\frac12\text{erf}(\frac{t-\mu}{\sigma\sqrt2}))(\frac{1}{\sqrt{2\pi\sigma_0^2}}\exp{-\frac{(\mu-\mu_0)^2}{2\sigma_0^2}})}{\int_{-\infty}^{\infty}(\frac12-\frac12\text{erf}(\frac{t-\mu}{\sigma\sqrt2}))\Biggl(\frac{1}{\sqrt{2\pi\sigma_0^2}}\exp{-\frac{(\mu-\mu_0)^2}{2\sigma_0^2}}\Biggr)d\mu}$$ The denominator is simply a scaling factor which can be ignored for the time. On qualitative analysis, the $\text{erf}$ term will scale the normal differentially for different values of $\mu$ resulting in an assymmetric skewed distribution finally. The exact scaling depends on the particular values.

Plot the numerator normalized for unit area to get the graph you seek.

$\endgroup$
4
  • 1
    $\begingroup$ Great answer, thanks! Is the $(1-\Phi(.))\cdot \phi(.)$ product ($\Phi$ and $\phi$ for cdf and pdf, respectively, ignoring normalizing constant) something that is common or has been used elsewhere? Clearly we now know it is asymmetric, but beyond that? There is no easy way to derive the mean of the posterior, is there? $\endgroup$
    – Nameless
    Aug 9 '18 at 16:42
  • 2
    $\begingroup$ While there is plenty of literature on erf, I couldn't find any place where $(1-\Phi)(\phi)$ is properly characterized. The integral in the denominator is addressed here and here, and this might crop up while getting the $E[\mu]$. Will post if I do manage to find a closed form expression for the mean. I am not sure if a simple formula like the one for the regular kind of $x$ is possible. $\endgroup$ Aug 9 '18 at 17:25
  • $\begingroup$ The integral in the links above is different from the one in our case, but a similar approach might be applicable here too. Couldn't find exactly this one though. $\endgroup$ Aug 9 '18 at 17:38
  • $\begingroup$ Alright, thanks. I think my original question can be considered answered. $\endgroup$
    – Nameless
    Aug 9 '18 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.