Or are there counterexamples? Especially some that are not quite artificial by e.g. taking the limit of a PDE series?
The short answer is, as noted by Henning, yes. If you have the analytical expression of a solution, then you have a solution. This is maybe the best scenario.
What you do not have is whether there are also other solutions to the problem. How can you be sure you have all possible solutions?
Often it is also difficult to see properties of solutions if the formula is complicated. But in most cases, even for simple ODEs, we are not able to write out the exact solution. It is more the lucky special case if you have one.