Suppose I define two functions of $x$ in terms of a convex function $f$ with a unique minimum $x_0$:
$$f_1(x) = 1 \times f(x)$$
$$f_2(x) = 2 \times f(x)$$
Suppose I wanted to minimize each of these functions numerically.
Clearly, the minima of both occur at the same $x_0$.
But if I tried to do gradient descent, I would be performing the updates
$$x \gets x - \lambda \nabla f_k(x)$$
and therefore step sizes would be larger for $f_2$ than $f_1$.
To me, this doesn't make sense. The step sizes should not depend on the scaling factor!
It therefore seems to imply that gradient descent should be normalized such that the step sizes have constant magnitude regardless of the gradient's magnitude.
So what justification is there for having the step size increase with the magnitude of the gradient when it's obvious that scaling the original function will change the gradient but not the location of its extrema?