Do we have any way of knowing if natural phenomena in the real world follow the "Lipschitz Condition"? Recently, I keep coming across terms containing "Lipschitz" pertaining to statistical models and machine learning. This includes terms such as  "p-lipschitz (rho), lipschitz convexity, lipschitz loss, lipschitz continuity, lipschitz condition, etc.
For example, the (famous) Gradient Descent algorithm that is used heavily in the field of machine learning to optimize the "loss functions" of Neural Networks, allegedly claims that it can provide a solution arbitrarily close to the true solution, provided an infinite number of iterations and that the "loss function" is "Lipschitz Continuous":



Although the mathematical definitions of the Lipschitz Condition are quite detailed and complicated, I think I was able to find a simple enough definition to understand for this question:



I am not sure if this is correct or not, but from an applied perspective : I have heard that the Lipschitz Condition is a desirable property for mathematical functions (and the gradients of these functions) because it suggests that "small changes in the input values of the function can not result in infinite values of the output of the same function".
My intuition is that if a function you are dealing with does not satisfy the Lipschitz Condition and can potentially grow in infinite size provided small changes in the inputs - this would suggest that this function has the ability to display very volatile, erratic and chaotic behaviors, making it fundamentally difficult unpredictable and unstable to deal with:



My Question: When it comes to complicated mathematical functions (e.g. functions that model natural phenomena in the real world), we can use the definitions of "convexity" (e.g. provided here) to find out if the function is convex or non-convex (i.e. if the function does not follow the required convexity conditions, it must be non-convex).
Can the same be said about real world functions obeying the Lipschitz Condition?
For example - if we look at the specific "loss function" being used in a specific neural network (e.g. specified number of layers, neurons, types of activation function, etc.), it might be possible to prove whether this "loss function" obeys the Lipschitz Condition.
But what about the original function that we are trying to approximate with the neural network itself?
At this point, are we just assuming that natural phenomena in the real world (e.g. ocean tides, economic patterns in the market, animal migration trajectories, etc.) are obeying the Lipschitz Condition - either in their su generis form, or for the sake of the machine learning algorithms to work?
Thanks!
 A: One condition where Lipschitz continuity naturally arises is if the function is differentiable with a bounded derivative. Let's say the function $f$ is continuous on $[a,b]$, differentiable on $(a,b)$ and the derivative is bounded: $\left|\frac{df}{dx}\right|\le M$. As per Mean Value Theorem:
$$f(b)-f(a)=\frac{df}{dx}(\xi)(b-a)$$
for some $\xi\in(a,b)$. Now, applying modulus and using the above condition of boundedness, we get:
$$|f(b)-f(a)|=\left|\frac{df}{dx}(\xi)\right||b-a|\le M|b-a|$$
Notice if the condition is satisfied on the whole $[a,b]$, then it is satisfied on any subinterval $[x,y]\subset [a,b]$ for the same $M$, so:
$$|f(y)-f(x)|\le M|y-x|$$
for all $x,y\in[a,b]$, which is the Lipschitz condition.

One special case of this is of the function $f$ that is continuously differentiable on $[a,b]$ because, well, $\frac{df}{dx}$ is then continuous and thus necessarily bounded.
If you can now argue that, in most applications of calculus in physics and natural sciences, all real functions of a real variable are not only continuously differentiable once, but are normally continuously differentiable infinitely many times, in fact they are analytic - then they will all be Lipschitz functions on any closed finite interval due to the previous statement.
