Consider the linear model $Y_i=X_i\beta_0+\epsilon_i$ with $\epsilon_i|X_i\sim N(0,\sigma^2_0)$, $\sigma^2_0>0$ and $\{Y_i, X_i\}_{i=1}^n$ i.i.d.

Let $\theta_0:=(\beta_0,\sigma^2_0)\in \Theta\subset \mathbb{R}^2$, $\Theta$ compact. So $\theta_0$ is the true parameter value, while $\theta$ denotes a generic parameter value.

Consider the (conditional) log likelihood of each observation $$m(Y_i, X_i;\theta):= -\frac{1}{2}\log(2\pi)-\log(\sigma)-\frac{1}{2\sigma^2}(Y_i-X_i\beta)^2$$

By WLLN, $\forall \theta\in \Theta$ $$ \frac{1}{n}\sum_{i=1}^n m(Y_i, X_i;\theta)\rightarrow_p E( m(Y_i, X_i;\theta)) $$

Question: How do I show that $E( m(Y_i, X_i;\theta))$ is uniquely maximised at $\theta_0$?

  • $\begingroup$ Are you assuming a random design? Are $X_i$ fixed or random? BTW, you are missing an $i$ in your first equation defining the model. $\endgroup$ – passerby51 Nov 30 '16 at 16:37

Let $(X,Y)$ be a generic version of $(X_i,Y_i)$, $v \equiv \sigma^2$ and let us show that $\ell(\theta;x) = E[ m(Y,X,\theta) | X = x]$ is maximized at $\theta_0$. We have

\begin{align*} \ell(\theta;x) &= - \frac12 \Big( \log v + \frac1v E(Y - x \beta)^2\Big) \\ &= - \frac12 \Big( \log v + \frac1v E(x\beta_0 + \epsilon - x \beta)^2\Big)\\ &= -\frac12 \Big( \log v + \frac1v E(x\beta_0 + \epsilon - x \beta)^2\Big)\\ &= - \frac12 \Big( \log v + \frac1v [x^2(\beta-\beta_0)^2 + \sigma_0^2]\Big) \end{align*} The unique maximizer over is clearly $\beta_0$ assuming that $x \neq 0$. Setting $\beta$ to this value, we need to minimize $v \mapsto \log v + \sigma_0^2/v$ which is uniquely maximized at $\sigma_0^2$.

Let $\ell(\theta) = E[m(Y,X,\theta)] = E[\ell(\theta;X)]$. Assuming that the distribution of $X$ puts positive mass on $X \neq 0$, the result follows by noting that $\ell(\theta;X) \le \ell(\theta_0;X)$ always and $\ell(\theta;X) < \ell(\theta_0;X)$ on some set of positive measure. (You could also directly compute $\ell(\theta)$ which amounts to changing $x^2$ to $E X^2$ in the expression above. This gives you the result assuming $E X^2 \neq 0$.)

  • $\begingroup$ Thanks. I have some questions. In the first set of derivations, when maximising wrto $v$, I am right if I say that we have to maximize $-\log(v)+\frac{\sigma^2_0}{v}$? Why did you write $\log(v)-\frac{\sigma^2_0}{v}$? $\endgroup$ – STF Nov 30 '16 at 17:48
  • $\begingroup$ @STF, that was a typo. I have updated, please check to see if that resolve it. We are maximizing the negative of what I wrote there which is equivalent to minimizing it. $\endgroup$ – passerby51 Nov 30 '16 at 17:54
  • $\begingroup$ @STF, BTW, I am also ignoring the constant term. (Let us say, my definition of $m$ does not include the constant.) $\endgroup$ – passerby51 Nov 30 '16 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.