For all $x\not =0 \in\mathbb{R}^m$ and $h\in\mathbb{R}^m$, who is $\xi'(x)\cdot h$? If $f,g:\mathbb{R}^m\rightarrow\mathbb{R}^n$ are differentiable with $f\in C^1$ and $\xi:\mathbb{R}^m\rightarrow \mathbb{R}$ where
$$\xi(x)=\langle g(x),\int_0^{|x|}f(tx)dt\rangle,$$
with $|x|=\sqrt{\langle x,x\rangle}$, then for all $x\not =0 \in\mathbb{R}^m$ and $h\in\mathbb{R}^m$, who is $\xi'(x)\cdot h$?
I don't know how to proceed with the second term. What I've done is:
$\forall x\not =0 \in\mathbb{R}^m$ and $h\in\mathbb{R}^m$,
$$\xi'(x)\cdot h=\lim_{w\to 0}{\frac{\xi(x+wh)-\xi(x)}{w}}=$$
$$=\lim_{w\to 0}\left\langle\frac{g(x+wh)-g(x)}{w},\frac{\int_{0}^{|x+wh|}f(tx+twh)dt-\int_{0}^{|x|}f(tx)dt}{w}\right\rangle=$$
$$=\left\langle g'(x)\cdot h,\lim_{w\to0}\left(\frac{\int_{0}^{|x+wh|}f(tx+twh)dt-\int_{0}^{|x|}f(tx)dt}{w}\right)\right\rangle$$
And after that, how could I establish that
$$\lim_{h\to 0}\frac{r_\xi(h)}{|h|}=\lim_{h\to 0}\frac{\xi(x+h)-\xi(x)-\xi'(x)\cdot h}{|h|}=0?$$
 A: Seems something is wrong.
Lets try to calculate the derivative of $\xi$ use inner product linearity.
$$\xi(x+wh)-\xi(x) = \langle g(t+wh), \int_{0}^{|x+wh|}f(t(x+wh))dt \rangle -\langle g(x), \int_{0}^{|x|}f(tx)dt \rangle$$.
Lets add and subtract $\langle g(x), \int_{0}^{|x+wh|}f(t(x+wh))dt\rangle$.
We will get
$$\xi(x+wh)-\xi(x) = \langle g(t+wh), \int_{0}^{|x+wh|}f(t(x+wh))dt \rangle - \langle g(x), \int_{0}^{|x+wh|}f(t(x+wh))dt\rangle+ \langle g(x), \int_{0}^{|x+wh|}f(t(x+wh))dt\rangle-\langle g(x), \int_{0}^{|x|}f(tx)dt \rangle$$
Combining the first two and the last twoyou will have
$$\xi(x+wh)-\xi(x) = \langle g(t+wh)-g(x), \int_{0}^{|x+wh|}f(t(x+wh))dt \rangle + \langle g(x), \int_{0}^{|x+wh|}f(t(x+wh))dt-\int_{0}^{|x|}f(tx)dt \rangle$$.
Dividing by $w$ and taking limit you'll get
$$\xi(x) \cdot h =  \lim_{w \to 0} \left(\langle \frac{g(t+wh)-g(x)}{w}, \int_{0}^{|x+wh|}f(t(x+wh))dt \rangle + \langle g(x), \frac{\int_{0}^{|x+wh|}f(t(x+wh))dt-\int_{0}^{|x|}f(tx)dt}{w} \rangle \right)$$.
So we get
$$ \xi'(x) \cdot h = \langle g'(x)\cdot h, \int_{0}^{|x|}f(tx)dt\rangle + \langle g(x), \int_{0}^{|x|}f'(tx)\cdot th \ dt \rangle$$.

The derivation of the integral part
We consider this
$$ \lim_{w \to 0} \left(\langle g(x), \frac{\int_{0}^{|x+wh|}f(t(x+wh))dt-\int_{0}^{|x|}f(tx)dt}{w} \rangle \right)$$
As the first component has no $w$, we let's just look at
$$\lim_{w \to 0}  \frac{\int_{0}^{|x+wh|}f(t(x+wh))dt-\int_{0}^{|x|}f(tx)dt}{w} =\lim_{w \to 0}  \frac{\int_{0}^{|x+wh|}f(t(x+wh))dt-\int_{0}^{|x|}f(t(x+wh))dt + \int_{0}^{|x|}f(t(x+wh))dt -\int_{0}^{|x|}f(tx)dt}{w} $$
Let's combine first two terms and last two terms.
$$\lim_{w \to 0}   \frac{\int_{|x|}^{|x+wh|}f(t(x+wh))dt + \int_{0}^{|x|}(f(t(x+wh))-f(tx))dt}{w} $$
In the second term interval of integration does not depend on $w$, so we can just differentiate under the sign of integration.
The first integral will surely converge to 0 as $w \to 0$, as the region of integration will become a point. But how fast?
As the integral of upper bound
We can rewrite
$$\lim_{w \to 0}   \frac{\int_{|x|}^{|x+wh|}f(t(x+wh))dt }{w}= f(|x|(x))\cdot h$$.
(So sorry I was wrong in my calculations).
So the correct answer should be
$$ \xi'(x) \cdot h = \langle g'(x)\cdot h, \int_{0}^{|x|}f(tx)dt\rangle + \left\langle g(x), \  f(|x|(x))\cdot h + \int_{0}^{|x|}f'(tx)\cdot th \ dt \right\rangle$$.
