0
$\begingroup$

Suppose we have a model $$y_i = g(x_i) + \epsilon_i, \quad i = 1, \ldots, n$$ where $y_i \in \mathbb{R}$ is a response variable, $x_i \in \mathcal{X} \subseteq \mathbb{R}^d$, and $\epsilon_i$ is the noise where $$E[\epsilon_i|x_i] = 0.$$

We approximate the function $g$ as follows: $$g(x_i) = x_i^T\beta + r_i$$ where $\beta \in \mathbb{R}^d$ is a vector of regression coefficients, and $r_i$ is the approximation error. We let $$A := E[x_ix_i^T]$$and $$\hat{A} := \frac{1}{n}\sum_{i=1}^n x_ix_i^T.$$

The least squares parameter $\beta$ is defined by

$$\beta := argmin_{b \in \mathbb{R}^d} E[(y_i - x_i^Tb)^2]$$ and the least squares estimator of $\beta$ is $$\hat{\beta}:= argmin_{b \in \mathbb{R}^d} \frac{1}{n}\sum_{i=1}^n (y_i - x_i^Tb)^2 = \hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_iy_i$$ Let $\alpha \in \mathbb{R}^d$ denote a vector such that $||\alpha|| = 1$, where $||\cdot||$ denotes the Euclidean norm.

I want to show that $$\sqrt{n}\alpha^T(\hat{\beta} - \beta) = \alpha^T \hat{A}^{-1}G_n[x_i(\epsilon_i + r_i)]$$

where $$G_n(f(w_i)) = \frac{1}{\sqrt{n}}\sum_{i=1}^n (f(w_i)) - E[f(w_i)]).$$

My attempt:

\begin{align*} \sqrt{n}\alpha^T(\hat{\beta} - \beta)&= \sqrt{n}\alpha^T\left(\hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_iy_i - \beta\right)\\ &= \sqrt{n}\alpha^T\left(\hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_i(x_i^T\beta + r_i + \epsilon_i) - \beta\right)\\ &= \sqrt{n}\alpha^T\left(\hat{A}^{-1}A\beta + \hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_i(r_i + \epsilon) - \beta\right)\\ &= \sqrt{n}\alpha^T\left(\hat{A}^{-1}\frac{1}{n}\sum_{i=1}^n x_i(r_i + \epsilon_i)\right)\\ &= \frac{1}{\sqrt{n}}\alpha^T\left(\hat{A}^{-1}\sum_{i=1}^n x_i(r_i + \epsilon_i)\right) \end{align*}

I'm stuck here and not sure how to show the rest because I'm stuck on the algebra. Any help would be appreciated.

$\endgroup$

1 Answer 1

0
$\begingroup$

Define $u_i:=\epsilon_i+r_i=y_i-x_i^{\top}\beta$. Then $$ \sqrt{n}(\hat{\beta}-\beta)=\left(\frac{1}{n}\sum_{i=1}^n x_ix_i^{\top}\right)^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n x_iu_i, $$ where $$ \mathsf{E}x_iu_i=\mathsf{E}[x_ig(x_i)+x_i\epsilon_i-x_ix_i^{\top}\beta]=0 $$ because $\beta=(\mathsf{E}x_ix^{\top})^{-1}\mathsf{E}x_ig(X_i)$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .