# How to find Frechet derivative of $f(x)=\|Ax-b\|^2$ at any $x^*$?

Given a real $$m \times n$$ matrix $$A$$ and $$b \in \mathbb{R}^m$$, let $$f(x)=\|Ax-b\|^2$$ for any $$x \in \mathbb{R}^n$$.

Find Frechet derivative of $$f(x)=\|Ax-b\|^2$$ at any $$x^*$$?

Actually I am wondering how to use the following to find the Frechet derivative, i.e., $$J$$:

$$\lim_{h \rightarrow 0} \frac{|f(x+h)-f(x)-Jh|}{\|h\|} =0$$

Another question is what the difference between what we would get from Frechet derivative and the gradient $$\nabla f(x)=2A^T(Ax-b)$$?

Please explain your reasons in detail, especially, what is the difference between gradient of $$f$$ and Frechet derivative. Also, explain when they might be identical.

• Write $f$ as a composition of an affine function and a bilinear function. – Will M. Dec 10 '18 at 4:05

If $$\mathrm{H}$$ is a Hilbert space, then every continuous linear function $$u:\mathrm{H} \to \mathbf{R}$$ can be represented by means of scalar product with respect to a unique vector, here denoted as $$x_u:$$
$$u(y) = (y \mid x_u).$$
Hence, if $$f:\mathrm{H} \to \mathbf{R}$$ is a differentiable function, then its derivative $$u = f'(a)$$ at $$a$$ is a continuous linear function. The vector $$x_u$$ is denoted $$\nabla f(a)$$ in this case. And we have the fundamental relation:
$$f'(a) \cdot h = (\nabla f(a) \mid h).$$
In regards to your particular $$f,$$ we can write $$f(x) = (Ax -b \mid Ax - b)$$ and by the products and chain rules, $$f'(x) \cdot h = (Ax - b \mid Ah) + (Ah \mid Ax - b) = 2(Ah \mid Ax - b).$$
If $$\mathrm{H} = \mathbf{R}^d,$$ and we are dealing with the standard Euclidean inner product, we can write further $$f'(x) \cdot h = (2A^\intercal (Ax - b) \mid h),$$ this signifies $$\nabla f(x) = 2A^\intercal (Ax - b).$$ Q.E.D.