I practice olympiad problems from books like Putnam and Beyond. Often I come across a problem that I simply can't solve. After $\sim30$ minutes of deep thinking it feels like I'm ramming my head into a brick wall, since I've exhausted all avenues of thought I am aware of. What should I do in these situations? Move on to another problem? Give up and see the answer? Or spend more time on it?
EDIT: To be fair, I never directly answered the question when I gave the four following points. I think there is a progression, though. Both (1) and (2) describe feelings you may have that suggest you should quit. My point is that neither by themselves point to giving up on a problem. For (2), I suggest you may need to think about why you want to solve a problem in the first place (note the philosophical nature of my answer to your question). But then, problem solving requires techniques, and asking when to give up is sort of like asking for more techniques. I suggest in (3) and (4) some techniques for progressing the process along that may help if you feel very stuck. Ultimately, you decide how hard you try and when to give up.
(1) That feeling of being stuck and/or somewhat frustrated is typical and part of what you sign up for when you try to solve hard problems. If you are training to be a mathematician you need to get used to this feeling. It's like being a soldier--it's just as much as a lifestyle change as knowing how to shoot guns. You need to cope with it and use it to your advantage. That you are asking this question shows you are starting to do this. I can't say I have it all figured out, but I will say that one of the most important things I learned in grad school is how to be at peace with this addicted/stubborn/frustrated feeling when working on a problem and not getting anywhere.
(2) Do not allow negative or self-derogatory thoughts. You will often think, "I am so stupid, other people smarter than me would have solved this problem ages ago." Or you might think, "Math is worthless, there's too much work and no reward." It is true, other people are smarter than you and will solve the problem faster than you--possibly. So what? Are you doing mathematics to impress other people? Or are you doing it for a job or for your own satisfaction or for your own education? Now what about math being not worth the effort? Maybe it's true. But there are different difficulties of problems. The Riemann hypothesis is probably too hard. For you, Putnam problems are probably just fine. So try to decide what difficulty is worth it to you. Don't avoid difficulty at all, since then there is no reward at all.
In any of these cases, they reveal a fundamental bias you have about math that should be addressed. Maybe you can't really convinced yourself that math is worth doing. Okay then, pick up something else instead. But just ask plenty of people here and they will have something convincing to say.
If you are truly becoming agitated, and can't beat these thoughts, the best thing to do is to distract yourself with something else. See (4).
(3) Find something of interest in the problem or tangentially related to the problem. When you are stuck, you often become bored of the problem. There is just nothing new you can see! If you could see something new, you wouldn't feel stuck. But have you considered changing the premises of the problem? Have you tried searching for examples or counter examples of the hypothesis? Do any mathematical techniques come to mind, even tangentially, when working on the problem? Think about these things instead. The idea is to find something easy to explore about the problem. Be creative and don't have the goal of solving the problem completely in mind. Have your enjoyment in mind.
(4) Sleep on it, eat lunch, go for a walk, or crumple up the problem statement and dig it out from the trash in a month. Often the ability to solve a certain problem is based on what you have already learned. If you haven't done the hard work of, say, learning a particular inequality derived from Holder's inequality, then it will be impossible to use it in any problem that requires it. Maybe you have learned it but not very well or a long time ago. In that case, you need to give your mind some time to bring it up to the front.
I think, thinking for 30 minutes is not too much. Try hard even for much longer time, couple of hours if possible ( A real challenging problem should at least take a day). Write down also the intermediate important conclusions, so that you don't have to discover them again and again. Personally I love to eat, drink and sleep with a problem, if I see some beauty in it, until there is light at the end of the tunnel.
It depends on the problem. Since you've tried a lot of problems already, I think it is not too hard to guess whether the solution is really completely out of your reach. In that case, either you can look at a solution if there is one, or you can simply put the problem aside and wait until you feel that it is within your reach. In the latter case, even years is normal. In my own experience there have been quite a number of such problems that I did not know how to tackle when I first encountered them, but after a few years I suddenly can make the necessary connections to solve them, sometimes even between fields of mathematics that previously did not seem related in that way. On the other hand, if you guess that the solution is somewhat within your reach, you may want to try for a few hours at a time, but with equally long breaks in-between. After a few sessions it will probably become clear if it wasn't indeed in your reach, but along the way you might have discovered other results, which would be nice and rewarding in themselves anyway.
30 minutes is nothing. Even for the imo there's supposed to be one and a half hours for each problem, although the last problem is supposed to take up more. For me it depends on how much I like the problem, with problems I don't like I give them 1 hour, however 1 hour of really thinking about it. And if I don't solve it after that I'll look at the solution. However if it's a problem I really like, I can think about it for weeks before thinking about looking for the solution. Something I do a lot of times is go into the unsolved section of mathlinks.ro and do those problems, this is really good for me because I don't have the temptation of giving up and looking the solution, and it's an even bigger challenge. In conclusion, at least 1 hour for me, with no upper bound depending on how much you like it.
In his book "Indiscrete Thoughts", the mathematician Gian-Carlo Rota describes another "Feynman method":
"Richard Feynman was fond of giving the following advice on how to be a genius. You have to keep a dozen of your favorite problems constantly present in your mind, although by and large they will lay in a dormant state. Every time you hear or read a new trick or a new result, test it against each of your twelve problems to see whether it helps. Every once in a while there will be a hit, and people will say: 'How did he do it? He must be a genius!'"
I'd like to add a bit to the other replies (in particular @nayrb's reply):
- Indeed when your wheels grind to a halt, stressing them even further would not yield much better results; it is a case of diminishing returns. Stop, rest, do something else for a while, even give yourself a rest for a couple of days before getting back to the problem.
- The brain does a lot of processing "offline" - when you're not actively engaging a problem. A lot of it in your sleep. Sometimes when you get back to a problem from last week you'll see that it's "so obvious" that "why was it a problem to begin with"?
- Our brains are very context-sensitive, you can see that in phenomena like priming and the "tetris effect". What this means for you is that you might become overly-focused on a single course of action or a single strategy of proof (or even a couple), and lose your "outsider" point of view - weigh too heavily on an analytic geometry strategy, for example, and at some point you start seeing everything through the "goggles" of analytic geometry. And the only way to ease that is taking some time off.
- And indeed don't get into a bad mood if you don't succeed for a while. Be reasonable about it: It's a problem, it requires tackling, and tackling takes time. And even if after much tackling and many a day of pondering you did not reach a solution, there's no shame in checking the answers. Not all problems are worth solving on their own merits - from some you'll learn more than from others, and after all it's all a matter of improving your abilities and having fun while doing it. If you come to a sense of exhaustion - not mental or emotional, but of the realm of possible solutions - then you've exhausted the benefit that you can get from a certain problem. And the same holds if there's still some benefit you can get but the process is too long and too frustrating to justify it. It's a matter of cost/benefit, and if there's another problem, the process of finding a solution to is just as fulfilling but more enjoyable or more conducive to your mathematical abilities - move on, it's no shame.
- Last but not least, remember that your question is just a special case of problem solving. You may find that some general reading on the subject is helpful both for your particular conundrum as well as for others that might arise in the future.
The answer for you changes with time.
This is much like the overused joke about the math professor working for $45$ minutes on a problem, looking up, and saying "it's trivial!" As you work more and more on math problems, you start to be able to work longer without becoming discouraged. For example, $30$ minutes may seem like forever now, but after working on a ton of (hard/stretching) problems, you'll probably think it seems small.
Always stretch/push yourself, and never lose track of why you're solving the problem.
Of course, if you mean "When to give up on a hard math problem, during competition for strategic reasons," that's a different problem entirely.