0
$\begingroup$

Suppose we are given a set of random observations $\{y_i,x_{i1},\dots,x_{ip}\}_{i=1}^N$. Based on these observations, we can form the multiple linear regression model in matrix form $$ Y_N = X_N\beta + \varepsilon_N, $$ where the error, conditioned on $X$, is normally distributed with mean zero and finite variance. I have used the subscript $N$ to explicitly indicate that we have used $N$ observations.

When we solve this model with least squares we find the estimated regression coefficient vector $\hat \beta_N$. We could also leave out the last observation and solve the model $$ Y_{N-1} = X_{N-1}\beta + \varepsilon_{N-1}, $$ to find the estimated regression coefficients $\hat \beta_{N-1}$.

The explicit expression for the estimated regression coefficients is well-known. We have $$ \begin{align} \hat \beta_N &= \beta_N + (X_N^TX_N)^{-1}X^T \varepsilon_N, \\ \hat \beta_{N-1} &= \beta_{N-1} + (X_{N-1}^TX_{N-1})^{-1}X^T \varepsilon_{N-1}. \end{align} $$ As $N$ gets large intuitively it seems there will be negligible difference between the estmiated regression coefficients $\hat \beta_N$ ad $\hat \beta_{N-1}$. But can we quantify this precisely? Is it possible to show that $E(N) := \hat \beta_N - \hat \beta_{N-1}$ converges to zero in probability or almost surely as $N \to 0$? And in particular, how fast it converges to zero?

$\endgroup$
1
  • $\begingroup$ I will comment also on the convergence in $L^p$ (your question on mathoverflow), if you edit the question accordingly. $\endgroup$
    – zhoraster
    Dec 14 '20 at 16:59
1
$\begingroup$

(From the discussion on MO) I guess that the subscript $N$ in $\beta_N$ is a lapse.

It is well know that, conditionally on $X_N$, $\hat \beta_N$ is unbiased and $\mathrm{Cov}(\hat \beta_N) = \sigma^2 \big(X_N^T X_N\big)^{-1}$.

Now in order to have some convergence, one naturally needs additional (to the usual Gauss–Markov) assumptions. Say, if $x_i$ are iid and square integrable, then $$ X_N^T X_N = \sum_{i=1}^N |x_i|^2 \sim n\,\mathrm{E}[|x_1|^2], n\to\infty, $$ almost surely. As a result, $\hat \beta_n \overset{\mathrm P}\longrightarrow \beta$, $n\to\infty$, moreover, $\sqrt{n} |\hat\beta_n - \beta|$ is bounded in probability. (So the convergence rate is loosely $1/\sqrt{n}$.)

$\endgroup$
3
  • $\begingroup$ Thanks for addressing this question. My central issue at the moment is explained in the comments of my latest post on stats.stackstackexhange. Essentially I need to know the order of magnitude of the moments $E[(\hat \beta - \beta)^k]$ for $k \ge 3$, as $n \to \infty$ because I am doing a second order Taylor expansion and I want to be sure it is safe to drop the third and higher order terms. $\endgroup$
    – sonicboom
    Dec 14 '20 at 17:17
  • $\begingroup$ It would be great if you could take a look at the stats.se post it if you get a chance, I cannot find anything in the literature that provides the order of magnitude of these moments. $\endgroup$
    – sonicboom
    Dec 14 '20 at 17:20
  • $\begingroup$ I'll have a look. $\endgroup$
    – zhoraster
    Dec 14 '20 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.