Asymptotic Normality of MLE when data is modelled with covariates Say I have data vector $X_1,\ldots,X_n$ which I want to model with some parametric distribution function $f(X_i;\theta,Z_i)$ and covariates $Z_i$. In this case, how can I prove the asymptotic normality of the maximum likelihood $\hat{\theta}$? 
Many proofs of MLE's asymptotic normality focus on the situation where $X_1,\ldots,X_n$ are i.i.d. (e.g., Asymptotic normal-behaviour of the MLE, question about proof.). Let $S_i(\theta)=\frac{\partial}{\partial \theta} \ln f(x_i,\theta)$, $S'_i(\theta)=\frac{\partial^2}{(\partial \theta)^2} \ln f(x_i,\theta)$,
$S_n(\theta)=\sum_{i=1}^n S_i(\theta)$ and $S'_n(\theta)=\sum_{i=1}^n S'_i(\theta)$. When $X_i$'s are i.i.d., the proof roughly takes the following steps:
(1) MLE is consistent $\hat{\theta}_n \rightarrow \theta$ where $\theta$ is the true value. The proof (e.g., http://ocw.mit.edu/courses/mathematics/18-443-statistics-for-applications-fall-2006/lecture-notes/lecture3.pdf) often uses LLN and $\ln f(X_i;\theta)$ needs to be i.i.d.
(2) $\sqrt{n}(\hat{\theta}_n-\theta) \approx \frac{\frac{1}{\sqrt{n}}S_n(\theta)}{\frac{1}{n}S'_n(\theta)}$. The approximation holds for large $n$ because of (1)
(3) The numerator converges in distribution to 
$$
N(0,\operatorname{Var}(S_i(\theta))) \tag{$*$}
$$ by CLT. 
(4) The denominator converges in probability to $E(S'_i(\theta))$ by LLN.
(5)$E(S'_i(\theta))=\operatorname{Var}(S_i(\theta))=I_1(\theta)$
(6) By Slutsky theorem and (2),(3),(4),(5), and letting Z is from $(*)$: 
$$\sqrt{n}(\hat{\theta}_n-\theta) \rightarrow E(S'_i(\theta))^{-1} Z \sim N(0,E(S'_i(\theta))^{-1}
\operatorname{Var}(S_i(\theta))
E(S'_i(\theta))^{-1})
=N(0,I_1(\theta)^{-1}).$$
However when the data is modelled with covariates, the data is no longer identically distributed. Therefore, the LLN and CLT used in (1), (3), (4) do not hold. Can anyone explain how to prove the asymptotic normality of MLE in this circumstance? 
 A: I guess, one way to convince myself for the asymptotic normality of MLE when $X_1,\cdots, X_N$ are independent but not identical is to use roughly the same arguments as i.i.d. scenario except the following two changes:
[Change 1] The classical i.i.d. CLT is replaced with Lyapunov CLT (http://en.wikipedia.org/wiki/Central_limit_theorem#cite_ref-6) which does not require identical observations (but independent is required). It states that under some conditions, if $Z_i \overset{ind}{\sim} (\mu_i,\sigma^2_i)$ then:
$$
\frac{1}{(\sum_{i=1}^{n}Var(Z_i))^{1/2}}\sum_{i=1}^{n} (Z_i-\mu_i) \overset{d}{\rightarrow} N(0,1)
$$
[Change 2] 
Let $Z_i \overset{ind}{\sim} (\mu_i,\sigma^2_i)$ and $c_i,i=1,2,\cdots$ be the sequence of constants.
Instead of using the regular SLLN, use arguments similar to WLLN with a condition that $\frac{\sum_{i=1}^n \sigma_i^2}{ (\sum_{i=1}^n c_i)^2}
\rightarrow 0 $ as $ n\rightarrow \infty$. 
By Chebychev's inequality for all $\epsilon$:
$$
Pr(|\frac{\sum_{i=1}^n Z_i}{\sum_{i=1}^n c_i}-\frac{\sum_{i=1}^n\mu_i}{\sum_{i=1}^n c_i}|\ge \epsilon)
\le 
\frac{Var(\frac{\sum_{i=1}^n Z_i}{\sum_{i=1}^n c_i}-\frac{\sum_{i=1}^n\mu_i}{\sum_{i=1}^n c_i})}{\epsilon^2}
=\frac{Var(\sum_{i=1}^n Z_i)}{(\sum_{i=1}^n c_i)^2\epsilon^2}=\frac{\sum_{i=1}^n \sigma_i^2}{  (\sum_{i=1}^n c_i)^2\epsilon^2}
$$
So if the condition is satisfied, then we have WLLN for non-identical scenario for the sequence of random variables $Z_1,Z_2,\cdots$. The LLN used in Step (1) and (4) of the proof for i.i.d. case above is replaced with this assuming the condition is satisfied. 
In summary, we can (roughly) show the asymptotic normality of MLE when the observations are not identically distributed taking the following steps:
(1) $\hat{\theta}\rightarrow \theta$ in probability under the condition discussed in [Change 2].
(2) The following approximation holds for large $n$ because of (1):
$$
[\sum_{i=1}^n Var(S_i(\theta))]^{1/2}
(\hat{\theta}-\theta)
\approx-
\frac{ 
\frac{1}{[\sum_{i=1}^n Var(S_i(\theta))]^{1/2}} S_n(\theta)
}{  
\frac{1}{[\sum_{i=1}^n Var(S_i(\theta))]} S_n'(\theta)
} 
=-
\frac{ 
\frac{1}{[\sum_{i=1}^n Var(S_i(\theta))]^{1/2}} \sum_{i=1}^n S_i(\theta)
}{  
\frac{1}{[\sum_{i=1}^n Var(S_i(\theta))]} \sum_{i=1}^n S'_i(\theta)
}
$$
(3) The numerator converses to $N(0,1)$ by Lyapunov CLT [Change 1].
(4) By LLN of [Change 2], for large $n$ the denominator is approximately $\frac{1}{\sum_{i=1}^n Var(S_i(\theta))} \sum_{i=1}^n E(S'_i(\theta))$
(5) 
$
Var(S_i(\theta)) = E(S'_i(\theta))= I_i(\theta)
$
for all $i$.
(6) By Slutsky theorem and (2),(3),(4),(5), for large $n$, 
$$
[\sum_{i=1}^n Var(S_i(\theta))]^{1/2}
(\hat{\theta}-\theta)\approx 
N(0,[\frac{ \sum_{i=1}^n E(S'_i(\theta))   }{\sum_{i=1}^n Var(S_i(\theta))} ]^{-2})
$$
Therefore:
$$
\hat{\theta}\approx
N(0,\frac{\sum_{i=1}^n Var(S_i(\theta))}{ [\sum_{i=1}^n E(S'_i(\theta))]^2 } )
=
N(0,\frac{1}{ \sum_{i=1}^n I_i(\theta) } )
$$
A: You can do this with central limit theorems for the non iid case. For example, check out "Estimation, inference, and specification analysis" book by halbert white. However easy way is to simply assume covariate distribution specifies a sequence of iid covariate patterns and this covariate distribution is not functionally dependent on theta
