Something like this used to be popular enough back in the nineteenth century (and maybe earlier): various scholarly bodies would pose prize problems and leading mathematicians (and maybe others? history doesn't say so much about that!) would submit "essays", i.e., research papers expressly directed at a particular topic or towards the solution of a particular problem. It is my understanding that they would usually award a prize to at least one person, even if the problem -- if there was a specific problem, which was sometimes not the case -- wasn't completely solved.
The two big examples I can think of off the top of my head are:
1) In 1883 Minkowski (at the age of 18!), together with H. Smith, won a prize set by the French Academy of Sciences for his essay on the theory of quadratic forms.
I should say that I don't know exactly what problem on quadratic forms, if any, the committee had in mind, and Minkowski and Smith's essays do rather different things. Of course, Minkowski's work at least solved a major problem -- the local-global principle for rational quadratic forms -- just maybe not the problem they specifically asked for! (And I think Smith's essay was pretty good too...)
2) In 1887, King Oscar II of Sweden, advised by Mittag-Leffler, established a prize for the solution of the three-body problem. The story of what happened is rather notorious. The following quote is taken from this wikipedia article, which summarizes it well:
In case the problem could not be solved, any other important contribution to classical mechanics would then be considered to be prizeworthy. The prize was finally awarded to Poincaré, even though he did not solve the original problem. One of the judges, the distinguished Karl Weierstrass, said, "This work cannot indeed be considered as furnishing the complete solution of the question proposed, but that it is nevertheless of such importance that its publication will inaugurate a new era in the history of celestial mechanics." (The first version of his contribution even contained a serious error; for details see the article by Diacu).
(In fact, if the popular books on the Poincaré Conjecture that I've read can be believed, the parenthetical sentence at the end is an understatement: apparently Poincaré's original paper was something of a mess.) $ $
I do not know of any 20th century or 21st century examples of prize essays like the above, but that doesn't mean they don't exist. The analogue in our day seems to be targeted grants, e.g. the DARPA debacle of a few years ago. But maybe it would be fun to revive the practice of old-timey prize essays? If anyone is reading this who has more money than s/he knows what to do with and aspires to be a philanthropist for mathematics, let me know! In fact, that's a good standing offer, whether you want to fund a prize essay or not. :)