# First order multivariate approximation

To demonstrate that $\nabla\!_{\hat{\boldsymbol u}}\,f(\boldsymbol{x}) \equiv \left \langle \hat{\boldsymbol u}, \nabla f(\boldsymbol{x}) \right \rangle$ I plug a first order expansion of $f(\boldsymbol{x}+t\boldsymbol{\hat{u}})$ into the definition of the directional derivative.

My point now is that I totally forgot from where the first order expansion comes from. I know that I can write

$$f(\boldsymbol{x}+t\boldsymbol{\hat{u}})=f(\boldsymbol{x})+\sum_{i=1}^N\frac{\partial f(\boldsymbol{x})}{\partial x_i}t\hat{u}_i+\mathcal{O}(t^2)$$

where $\boldsymbol{x},\hat{\boldsymbol{u}}\in\mathbb{R}^N$, $f:\mathbb{R}^N\rightarrow\mathbb{R}$, $\hat{u}_i$ is the $i$-th component of $\hat{\boldsymbol{u}}$, $t\in\mathbb{R}$ and $\mathcal{O}$ is the order operator.

Which reminds me the mono-dimensional Taylor expansion (easily derivable), but perhaps I'm going too far and there's no need to trouble him in this case.

So, where does this first order expansion come from?
(And would it be easy to derive the second and higher orders as well?)

-

The map $f:\mathbb{R}^{n}\rightarrow \mathbb{R}$ is continuously differentiable if we have $\frac{\partial f}{\partial x_{i}}$ defined continuously everywhere. And similarly you can define second order derivative, etc (which is the Hessian). A rigorous proof of the general case can be found in serious analysis books like Zurich(Mathematical Analysis II).

But for your question, there is a simpler answer by converting it to the 1 dimension case, such that $f$ becomes a map from $\mathbb{R}\rightarrow \mathbb{R}$ with $f(t)=f(x+t e_{0})$, and $|e_{0}|=1$. The rest simply follows by chain rule. You can find a derivation of it (in case you cannot do it yourself) in Frielander&Moshi's book Introduction to the theory of distributions. I think it is either Chapter 1 or Chapter 2 where they introduce the multi-index.

-
For the 1-dimensional case I can do my polynomial expansion demonstration in $x_0$ by computing $f(x_0)$, $f'(x_0)$ and so forth, ending up to the known $\mathcal{T}_{x_0}f(x)$ Taylor serie. Here I was actually asking if someone can provide me the equivalent (at least for the linear term) for the multivariate case. –  Atcold Sep 11 '13 at 3:37
$f$ is function of $x$ not $t$ or are you trying to convey something different? Moreover, could you clarify the "The rest simply follows by chain rule." statement of yours? –  Atcold Sep 11 '13 at 3:42
@Atcold: Consider $f(t)=f(x_{0}+e_{0}t)$ as a function of $t$, with $x_{0}$ and $e_{0}$ fixed. –  Bombyx mori Sep 11 '13 at 4:59
@Atcold: What is the good of a correct formula if you cannot derive it? Try to finish the proof and see how it might extend to the second order Taylor expansion, then by generalizing you get the formula you wanted in general. –  Bombyx mori Sep 11 '13 at 5:01
Let's $f$ be defined on $A$, open subset of $\mathbb{R}^N$, $\boldsymbol{x}_0$ fix point of $A$ and $f:A\rightarrow\mathbb{R}$. $$\bar{f}(\boldsymbol{x})=L(\boldsymbol{x}-\boldsymbol{x}_0)+q$$ is the linear approximant in $\boldsymbol{x}_0$ --- where $L\in\mathcal{L}(\mathbb{R}^N,\mathbb{R})$ (which means $L\in\mathbb{R}^{1\times N}$), $q\in\mathbb{R}$ --- if \begin{align} &\bar{f}(\boldsymbol{x}_0)=f(\boldsymbol{x}_0)\\ &f(\boldsymbol{x})=\bar{f}(\boldsymbol{x})+\varepsilon(\boldsymbol{x})\left \| \boldsymbol{x} -\boldsymbol{x}_0\right \|,\quad\varepsilon(\boldsymbol{x})\rightarrow 0,\boldsymbol{x}\rightarrow \boldsymbol{x}_0 \end{align} Then, if it $\exists$, $$f(\boldsymbol{x})=f(\boldsymbol{x}_0)+L(\boldsymbol{x}-\boldsymbol{x}_0)+\varepsilon(\boldsymbol{x})\left \| \boldsymbol{x} -\boldsymbol{x}_0\right \|,\quad\varepsilon(\boldsymbol{x})\rightarrow 0,\boldsymbol{x}\rightarrow \boldsymbol{x}_0,\quad \mathrm{d}f(\boldsymbol{x}_0):=L=\boldsymbol{a}^\top,\boldsymbol{a}\in\mathbb{R}^N$$ whereas $\mathrm{d}f(\boldsymbol{x}_0)$ is the differential of $f$ in $\boldsymbol{x}_0$. Therefore we have that \begin{align} \frac{\partial f}{\partial \boldsymbol{v}}(\boldsymbol{x}_0)&=\lim_{t\rightarrow 0}\frac{f(\boldsymbol{x}_0+t\boldsymbol{v})-f(\boldsymbol{x}_0)}{t}=\lim_{t\rightarrow 0}\frac{f(\boldsymbol{x}_0)+L(t\boldsymbol{v})+\varepsilon(\boldsymbol{x}_0+t\boldsymbol{v})\left \|t\boldsymbol{v}\right \|-f(\boldsymbol{x}_0)}{t}\\ &=\lim_{t\rightarrow 0}\frac{t}{t}L(\boldsymbol{v})+\lim_{t\rightarrow 0}\frac{|t|}{t}\varepsilon(\boldsymbol{x}_0+t\boldsymbol{v})\left \|\boldsymbol{v}\right \|=L(\boldsymbol{v}) \end{align} $$\frac{\partial f}{\partial x_i}(\boldsymbol{x}_0)=L(\boldsymbol{\hat{e}}_i)=\left \langle \boldsymbol{a}, \boldsymbol{\hat{e}}_i\right \rangle=a_i$$ $$\Rightarrow L(\boldsymbol{u})=(\mathrm{d}f(\boldsymbol{x}_0))(\boldsymbol{u})=\frac{\partial f}{\partial x_1}(\boldsymbol{x}_0)u_1+\cdots +\frac{\partial f}{\partial x_N}(\boldsymbol{x}_0)u_N,$$ Summarising, we have that $$\nabla f(\boldsymbol{x}_0):=\left (\frac{\partial f}{\partial x_1}(\boldsymbol{x}_0),\cdots ,\frac{\partial f}{\partial x_N}(\boldsymbol{x}_0) \right )^\top$$ $$(\mathrm{d}f(\boldsymbol{x}_0))=L(\boldsymbol{u})=\left \langle \nabla f(\boldsymbol{x}_0), \boldsymbol{u}\right \rangle$$ $$\frac{\partial f}{\partial \boldsymbol{v}}(\boldsymbol{x}_0)=L(\boldsymbol{v})=\left \langle \nabla f(\boldsymbol{x}_0), \boldsymbol{v}\right \rangle,\quad\left \| \boldsymbol{v} \right \|=1$$ $$f(\boldsymbol{x})=f(\boldsymbol{x}_0)+\left \langle \nabla f(\boldsymbol{x}_0), \boldsymbol{x}-\boldsymbol{x}_0\right \rangle+\varepsilon(\boldsymbol{x})\left \| \boldsymbol{x} -\boldsymbol{x}_0\right \|,\quad\lim_{\boldsymbol{x}\rightarrow \boldsymbol{x}_0}\varepsilon(\boldsymbol{x})=0\quad\square$$