Via the Halting Problem and Incompleteness Theorem, in some sense there isn't a perfect answer to your question. That said, there might be lots of good answers depending on your goals. I'll answer the related question, "How do you decide [when] to give up on such a problem?"
- Never. Pause problems rather than throwing them in the bin. People learn over time, and there is no telling what new fact, tidbit, or insight is going to make the difference in your mental model of a previously hard problem.
- When you find something more productive. Talking to colleagues or attending conferences exposes oneself to new ideas. When combined with your own personal worldview and experience, this can open up a flurry of productive research topics. If your goal is to quickly churn out new work, these paths where you can see closer to the end might be a better idea than your current roadblock.
- Whenever you find something more interesting. Life isn't all about productivity. Study what makes you happy, and when that fails study something else. Eventually you'll find a project that is also productive, and you won't have beaten yourself up to find it. Most of the math professors I know use some variation of this strategy.
- Never. I know I listed this twice, but this can be important for another reason. Work on hard problems often turns up other intriguing results. Another answer by @Arthur mentioned Fermat's Last Theorem, and that's a great example of entire fields of math being discovered just to create a single proof. I have a pet number theory problem I reexamine periodically that always has something new going on.
- Often. Staring at the same problem without progress is a waste of time and drains your mental resources. Unless you're really making headway toward something, take a break every half hour or so and go for a walk or find somebody to talk to.
- When progress stalls. On too many occasions to count, problems I thought would be fruitful did not have clean answers that I could find. After exhausting all available resources, I filed away my work to pick them up at a later date. With additional knowledge and experience I've since solved some of them, proven others can't be solved, and made absolutely no headway with a substantial majority of those files.
Everybody has their own tolerance for work without results, but mine is very low if I'm not at least learning something new or making some small discovery in the process. Other people suggest having a single project at a time (or a major project with large impact and a minor project to get a guaranteed paper out), but I think overall productivity is higher in creative disciplines like math when you can switch to whichever idea you currently have the most insights into, at the expense of each paper taking longer from start to finish.