# Understanding a Taylor expansion in probability

I initially posted this on stats.stackexchange but received no feedback. Maybe this is better suited for math.stackexchange since the question is mainly about Taylor expansion with random variables.

I'm trying to understand the proof of an expression for the asymptotic bias in local polynomial regression of degree $p\ge0$.

Specifically, I'm distraught with equation $(3.59)$ on page 102 of this book. This is part of the proof of Theorem 3.1 on page 62. (You can read the full passages by clicking on the snippets I have linked to.)

Here's the setup. Let $(X_1, Y_1),\ldots,(X_n,Y_n)$ be iid taking values in $\mathbb R^2,$ let $X_1$ have density $f.$ Let $m(x)=E(Y|X=x)$ and let $K$ be a symmetric kernel with bounded support, let $K_h(t) = K(t/h)/h,$ where $h$ is the bandwidth. Write $$\mathbf X=((X_i-x_0)^j)_{i=1,\ldots,n \atop j=0,\ldots,p}, \mathbf W = \operatorname{diag}(K_h(X_1-x_0),\ldots,K_h(X_n-x_0)),\\ \mathbf y=(Y_1\ldots,Y_n)^T, \mathbf m=(m(X_1),\ldots,m(X_n))^T.$$

Then the conditional bias of the local polynomial estimator $\hat\beta=(\mathbf X^T \mathbf W \mathbf X)^{-1}\mathbf X^T \mathbf W \mathbf y$ is $$\operatorname{Bias}(\hat\beta|(X_1,\ldots,X_n))=(\mathbf X^T \mathbf W \mathbf X)^{-1}\mathbf X^T \mathbf W \mathbf r =: S_n^{-1}\mathbf X^T \mathbf W \mathbf r,$$ where $\mathbf r = \mathbf m-\mathbf X \beta,\, \beta=(m(x_0),\ldots,m^{(p)}(x_0)/{p!})^T.$

Assume that $m^{(p+1)}(\cdot)$ is continuous in a neighborhood of $x_0.$ Fan writes on page 102:

By using the Taylor expansion the conditional bias $S_n^{-1}\mathbf X^T \mathbf W \mathbf r$ of $\hat\beta$ can be written as $$S_n^{-1}\mathbf X^T \mathbf W \Bigl[\beta_{p+1}(X_i-x_0)^{p+1}+o_P\left\{(X_i-x_0)^{p+1}\right\}\Bigr]_{1\le i\le n}$$

I don't understand what is meant by $o_P\left\{(X_i-x_0)^{p+1}\right\}$ in this context. I know that the usual definition is that it's a term which converges in probability to zero even after dividing by $(X_i-x_0)^{p+1}.$ But what converges in probability here? Is it meant that this holds as $n\to\infty$?

I tried writing everything out using Lagrange remainder, but as as @TedShifrin pointed out, the Lagrange remainder term should have an exponent of p+2, not p+1. But $m$ is only $p+1$ times differentiable. So maybe the Lagrange-remainder is not as helpful here as I thought. Peano-remainder yields \begin{align} \mathbf r &= \mathbf m-\mathbf X \beta \\ &= \Biggl[\sum_{l=0}^{p+1} \frac{m^{(l)}(x_0)}{l!} (X_i-x_0)^{l} + (X_i-x_0)^{p+1}g_i(X_i-x_0)\\ &\quad\quad- \sum_{l=0}^{p} \frac{m^{(l)}(x_0)}{l!} (X_i-x_0)^{l}\Biggr]_{1\le i\le n}\\ &= \left[\frac{m^{(p+1)}(x_0)}{(p+1)!} (X_i-x_0)^{p+1} + (X_i-x_0)^{p+1}g_i(X_i-x_0)\right]_{1\le i\le n} \end{align} For a function $g_i$ with $g_i(t)\to 0$ as $t\to0.$ If the $X_i$ were not random we would write $(X_i-x_0)^{p+1}g_i(X_i-x_0)=o((X_i-x_0)^{p+1})$ as $X_i-x_0\to0.$ But since the $X_i$ are random, in what sense is $$(X_i-x_0)^{p+1}g_i(X_i-x_0)=o_P((X_i-x_0)^{p+1})?$$

• The Lagrange remainder term should have an exponent of $p+2$, not $p+1$. Then it'll all be $o((X_i-x_0)^{p+1})$. – Ted Shifrin Dec 8 '17 at 19:09
• @TedShifrin thank you, I edited my question to reflect this. Unfortunately $m$ is only $p+1$ times differentiable. In any case, I'm still unsure what is meant by $o_P((X_i-x_0)^{p+1})$ in this context, because sending $n\to \infty$ does nothing to $X_i-x_0.$ Is perhaps something like $$(X_i-x_0)^{p+1}g_i(X_i-x_0)=o_P((X_i-x_0)^{p+1}) \text{ as }X_i \overset{pointwise \atop for\,all\,\omega}{\to}x_0$$ meant? – Epiousios Dec 8 '17 at 20:21
• You don't need the Lagrange remainder. If a function is $k$ times differentiable at $x_0$, then the error between the function and its $k$th degree Taylor polynomial is still $o((x-x_0)^k$. This follows, for example, from L’Hôpital’s rule. – Ted Shifrin Dec 8 '17 at 20:30
• Right, but in what sense is it $o_{P}((X_i-x_0)^{p+1})?$ ($o_P$, Not $o$) – Epiousios Dec 8 '17 at 21:57
• I can't answer this, as I would have to spend a week trying to understand the probability context, and you've linked only to tiny little snippets, not even a readable passage. – Ted Shifrin Dec 8 '17 at 22:03