1
$\begingroup$

Here is the exact problem:

Suppose that the random variables $Y_1,\ldots,Y_n$ satisfy $$ Y_i=\beta x_i+\varepsilon_i, \quad i=1,\ldots,n. $$ where $x_1,\ldots,x_n$ are fixed constants, and $\varepsilon_1,\ldots,\varepsilon_n$ are i.i.d random variables with $\mathrm{Normal}(0,\sigma^2)$. Given data observations, denoted as $D$, i.e., $D=\{x_i,y_i\}^n_{i=1}$, answer the following question through calculation.

Assume $\sigma^2$ is known and assume the prior distribution of $B$ follows normal distribution, i.e., $\beta\sim \mathrm{Normal}(B_0,\sigma_o^2)$, where $B_0$ and $\sigma_0^2$ are known constants. Determine the marginal posterior distribution of $B$, i.e. $B\mid D$.

Here is my work so far:

Thus, the pdf of the priori distribution is:

$$\pi(\beta)=\frac{1}{\sqrt{2\pi}\sigma_0}e^{\frac{-(\beta-\beta_0)^2}{2\sigma_0^2}}$$

Next, I need to find the distribution of the $Y_i$ random variable.

$$\operatorname{E}[Y_i]=\operatorname{E}[\beta x_i+\varepsilon_i]=\beta x_i+0=\beta x_i$$

$$\operatorname{Var}[Y_i]=\operatorname{Var}[\beta x_i+\varepsilon_i]=\sigma^2$$

Thus, $Y_i \sim \mathrm{Normal}(\beta x_i, \sigma^2)$

Now, here is where I need help.

I know a sufficient statistic for the mean of a normal distribution when the variance is known is the sample mean.

Thus, having $\overline{x}=\frac{1}{n}\sum^n_{i=1}(\beta x_i)$ the likelihood becomes:

$$f(\overline{x}\mid \beta x_i)=\frac{1}{\sqrt{2\pi}\sigma}e^{\frac{-(\overline{x}-\beta x_i)^2}{2\sigma^2}}$$

So, I am wondering If my set up of the likelihood is correct. I know the posterior distribution is typically represented by $f(\bf{x}\mid \theta) \propto f(\theta \mid \bf{x}) \cdot \pi(\theta)$. In this representation notice that the parameter $\theta$ is the same throughout, but in the way I set up the problem it is not due to $\beta x_i$ and $\beta$. So, I'm wondering if any one can shed some light on how I need to represent the liklihood.

$\endgroup$
  • $\begingroup$ It's not correct since you have $n$ different normal distributions --- one for each separate value of $i$ in the range $1,\ldots,n$. Maybe I'll say more below. $\qquad$ $\endgroup$ – Michael Hardy Jul 14 '16 at 18:16
  • $\begingroup$ It would defiantly be appreciated. $\endgroup$ – kdrozd Jul 14 '16 at 18:42
1
$\begingroup$

Your proposed sufficient statistic assumes all $n$ of the normal distributions are the same, but they're not.

The conditional distribution of $Y_i$ given $B$ is $\mathrm{Normal}(Bx_i, \sigma^2 )$. The likelihood is $$ L(\beta) = \frac 1 {\sigma^{2n}} \exp\left( \frac{-1}{2\sigma^2} \sum_{i=1}^n ( y_i - \beta x_i )^2 \right). $$ This admits further simplification by doing some algebra, and that can identify a sufficient statistic.

The least-squares estimator of $\beta$ can be shown to be $$ \widehat\beta = \frac{\sum_i x_i y_i}{ \sum_i x_i^2}. $$ Then we have \begin{align} & \sum_{i=1}^n ( y_i - \beta x_i )^2 = \sum_i \Big( (y_i - \widehat\beta x_i) + (\widehat\beta x_i - \beta x_i)\Big)^2 \\[10pt] = {} & \left( \sum_i (y_i - \widehat\beta x_i)^2 \right) + \underbrace{2(\widehat \beta - \beta) \sum_i (y_i-\widehat\beta x_i) x_i}_\text{Show that this middle sum is $0$.} + \left(\widehat\beta-\beta\right)^2 \sum_i x_i^2. \end{align} You should ultimately end up with $\widehat\beta$ as the sufficient statistic.

$\endgroup$
  • $\begingroup$ Thank you a lot. If you could also add in how to get the posteriori from here that would help me a lot too. $\endgroup$ – kdrozd Jul 14 '16 at 20:20
  • $\begingroup$ Never mind, I got it. $\endgroup$ – kdrozd Jul 14 '16 at 23:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.