Given a real $m \times n$ matrix $A$ and $b \in \mathbb{R}^m$, let $f(x)=\|Ax-b\|^2$ for any $x \in \mathbb{R}^n$.

Find Frechet derivative of $f(x)=\|Ax-b\|^2$ at any $x^*$?

Actually I am wondering how to use the following to find the Frechet derivative, i.e., $J$:

$$\lim_{h \rightarrow 0} \frac{|f(x+h)-f(x)-Jh|}{\|h\|} =0$$

Another question is what the difference between what we would get from Frechet derivative and the gradient $\nabla f(x)=2A^T(Ax-b)$?

Please explain your reasons in detail, especially, what is the difference between gradient of $f$ and Frechet derivative. Also, explain when they might be identical.

  • $\begingroup$ Write $f$ as a composition of an affine function and a bilinear function. $\endgroup$ – Will M. Dec 10 '18 at 4:05

If $\mathrm{H}$ is a Hilbert space, then every continuous linear function $u:\mathrm{H} \to \mathbf{R}$ can be represented by means of scalar product with respect to a unique vector, here denoted as $x_u:$

$$u(y) = (y \mid x_u).$$

Hence, if $f:\mathrm{H} \to \mathbf{R}$ is a differentiable function, then its derivative $u = f'(a)$ at $a$ is a continuous linear function. The vector $x_u$ is denoted $\nabla f(a)$ in this case. And we have the fundamental relation:

$$f'(a) \cdot h = (\nabla f(a) \mid h).$$

In regards to your particular $f,$ we can write $f(x) = (Ax -b \mid Ax - b)$ and by the products and chain rules, $$f'(x) \cdot h = (Ax - b \mid Ah) + (Ah \mid Ax - b) = 2(Ah \mid Ax - b).$$

If $\mathrm{H} = \mathbf{R}^d,$ and we are dealing with the standard Euclidean inner product, we can write further $f'(x) \cdot h = (2A^\intercal (Ax - b) \mid h),$ this signifies $\nabla f(x) = 2A^\intercal (Ax - b).$ Q.E.D.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.