In the realm of statistical modelling, creating a statistical model with respect to some data involves optimizing some mathematical function (e.g. Loss Function, OLS Equation, Maximum Likelihood Equation, etc. ) : Determining the final beta regression coefficients involves solving an optimization equation.
From what I see:
Sometimes, the equation to be optimized has a "closed form solution". For example, estimating the coefficients for a Linear Regression model involves optimizing an OLS or an MLE equation - but this optimization equation has a "closed form solution". This means that we can directly estimate the regression coefficients and do not require an iterative optimization algorithm. I have noticed that many of these instances where the optimization problem has a "closed form solution", the equation (i.e. the mathematical function being optimized) is usually "Convex".
However, in most cases these optimization equations do not have "closed form solutions" (e.g. Neural Networks, Logistic Regression, etc.). In these cases, an iterative optimization algorithm (e.g. Gradient Descent) has to be used to estimate the model parameters (e.g. weights of a Neural Network). I have also noticed that when this is the case, the optimization equation (i.e. the mathematical function being optimized) is usually "Non-Convex".
I understand that optimizing some Convex Functions have "closed form solutions" while other Convex Functions do not have "closed form solutions".
- Can a mathematical problem that is Non-Convex (i.e. a Non-Convex Function) ever have a "closed form solution"? Or is this never the case?