I am not sure if this question make sense mathematically, so please bear with my ignorance. This is an extension to the question in the link:
It seems in many engineering problem when we look for an optimal solution we work on normed vector space and it is complete under Lp norm. My question is how to improve convergence with less or limited data? It appears to me that since our metric space is complete the convergence issue is related to the structure of space. What I mean is, we may not able to get faster convergence with limited data if we work on linear vector space.
Is it possible that the problem may have a better solution in some function space which is not linear?