# Least squares estimation for linear regression model with random design

Suppose we have a model $$y_i = g(x_i) + \epsilon_i, \quad i = 1, \ldots, n$$ where $$y_i \in \mathbb{R}$$ is a response variable, $$x_i \in \mathcal{X} \subseteq \mathbb{R}^d$$, and $$\epsilon_i$$ is the noise where $$E[\epsilon_i|x_i] = 0.$$

We approximate the function $$g$$ as follows: $$g(x_i) = x_i^T\beta + r_i$$ where $$\beta \in \mathbb{R}^d$$ is a vector of regression coefficients, and $$r_i$$ is the approximation error. We let $$A := E[x_ix_i^T]$$and $$\hat{A} := \frac{1}{n}\sum_{i=1}^n x_ix_i^T.$$

The least squares parameter $$\beta$$ is defined by

$$\beta := argmin_{b \in \mathbb{R}^d} E[(y_i - x_i^Tb)^2]$$ and the least squares estimator of $$\beta$$ is $$\hat{\beta}:= argmin_{b \in \mathbb{R}^d} \frac{1}{n}\sum_{i=1}^n (y_i - x_i^Tb)^2 = \hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_iy_i$$ Let $$\alpha \in \mathbb{R}^d$$ denote a vector such that $$||\alpha|| = 1$$, where $$||\cdot||$$ denotes the Euclidean norm.

I want to show that $$\sqrt{n}\alpha^T(\hat{\beta} - \beta) = \alpha^T \hat{A}^{-1}G_n[x_i(\epsilon_i + r_i)]$$

where $$G_n(f(w_i)) = \frac{1}{\sqrt{n}}\sum_{i=1}^n (f(w_i)) - E[f(w_i)]).$$

My attempt:

\begin{align*} \sqrt{n}\alpha^T(\hat{\beta} - \beta)&= \sqrt{n}\alpha^T\left(\hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_iy_i - \beta\right)\\ &= \sqrt{n}\alpha^T\left(\hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_i(x_i^T\beta + r_i + \epsilon_i) - \beta\right)\\ &= \sqrt{n}\alpha^T\left(\hat{A}^{-1}A\beta + \hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_i(r_i + \epsilon) - \beta\right)\\ &= \sqrt{n}\alpha^T\left(\hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_i(r_i + \epsilon_i)\right)\\ &= \frac{1}{\sqrt{n}}\alpha^T\left(\hat{A}^{-1}\sum_{i=1}^n x_i(r_i + \epsilon_i)\right) \end{align*}

I'm stuck here and not sure how to show the rest because I'm stuck on the algebra. Any help would be appreciated.

Define $$u_i:=\epsilon_i+r_i=y_i-x_i^{\top}\beta$$. Then $$\sqrt{n}(\hat{\beta}-\beta)=\left(\frac{1}{n}\sum_{i=1}^n x_ix_i^{\top}\right)^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n x_iu_i,$$ where $$\mathsf{E}x_iu_i=\mathsf{E}[x_ig(x_i)+x_i\epsilon_i-x_ix_i^{\top}\beta]=0$$ because $$\beta=(\mathsf{E}x_ix^{\top})^{-1}\mathsf{E}x_ig(X_i)$$.