What are/ were some of your good mathematician's study habits that you found really worked for you? I'm a CS major at a respected school and have a solid GPA... However, I definitely lack when it comes to getting great grades in relatively higher level maths (300 and 400 level). I feel that even though I study a lot, I perhaps don't study effectively enough. I would like to ask for any tips/ tricks as to how you keep yourself focused, motivated etc that could perhaps help me or others. Thanks in advance.
Here are some suggestions:
In addition, I would recommend learning a Computer Algebra System (like Mathematica) and learn to explore problems and find various avenues to solve problems.
I hope that gives you some ideas and things to research further.
Let me add to the excellent advice from both @Amzoti and @user33263.
As you move on to higher level courses, the maths gets conceptually more difficult. Especially in books with a more traditional flavour, you can be thrown a stack of definitions with -- apparently -- not much motivation. You can be wondering "why is this notion important?", "what is the interest of that theorem?". And even if, with effort, you can tackle the exercises and jump through the hoops of showing that so-and-so follows from such-and-such, you can still initially be rather baffled by what's going on. (I found this, for example, when I recently started trying to teach myself some category theory.)
If you work at it, with luck, you should get more of the hang of things. But you can speed up the process considerably if you look at a lot more than one presentation of the basic ideas. If you are set a course text, the lecturer will of course think the book has its virtues. But there will be a lot of other texts out there with rather different virtues. Look at how they introduce basic ideas, motivate basic definitions, explain proof ideas, explain why they introduce the topics they do in the order they do, etc. etc. (That doesn't mean reading them all with the same detailed attention you have to give to your course text.)
You learn the geography of your city, learn your way around, by walking different routes, coming at places from different directions. Similarly, you should learn mathematical geography (so to speak) by coming at things from the different angles which a selection of different (or different enough!) texts will provide. That will help you feel more at home in a new area quicker, and (hopefully) engender a richer understanding of what's going on than you are likely to get from sticking to just one or two textbooks.
Here's something I didn't do until grad school, but wish I had started earlier. Depending on my schedule each quarter I find one hour every single morning that I set aside for "basics."
I find that I absolutely must specify this hour for that and only that if I'm going to psychologically get myself to do this. Otherwise I'll just keep putting it off for later and later in the day until I just don't get around to it. Everything else will seem more important, but I've found this time to be extremely useful.
Here's what I use the time for. Take a textbook on a subject you're learning (presumably the assigned textbook, but certainly others work to get a variety of perspectives). Start going through it. By going through it I mean you have a huge stack of paper (usually from the recycling next to a public printer). You read the book, and after every single sentence make sure you can justify why it is true.
Paragraphs from the book will sometimes transfer to many pages of your own writing. Also, don't let the topics get too fancy. This time is set aside for basics.
What I find odd in math compared to practically everything else is that this type of thing isn't emphasized. Even the top musicians in the world set aside time for scales everyday. It is easy to forget while struggling to play a hard piece of music that the basics are what makes it easy to put together (what am I talking about again?).
Maybe everyday is excessive for an undergrad class, but I think a few times a week will really get your brain in gear to understand the harder topics, and it will certainly help you fill in the details of proofs on quizzes/tests if you've already carefully thought through why the steps of the big theorems are true.
I am also from CS background, and now student of mathematics, initially I too struggled alot as I have 0 formal maths background and also everything is self study.
As far as my experience says,
All the best.
You should be pleased that you can see where you stand and that you don't delude yourself.
My suggestion (not unique) is to try to explain any process you are engaged with - a concept, a theorem, a h.w. problem - so that you can convince yourself as well as be able to explain it to anyone. To make the point, you may even want to present some necessary background material and elaborate on the relevant definitions.
With your sense of self-honesty you will easily be able to see where you may be stuck or have a hole in your grasp of the material.
You can do this any time, taking a walk (I even do it while swimming laps), etc.
I think working out like this can give you a sense of confidence and, as you resolve issues, a good grasp of the material.
You asked why you are not getting good grades in higher level math course. Start by looking at how the grades are actually given for these courses. It all based on an exam or split with weekly course work? Are the types of questions different that on the lower level courses that you did well at? For example do they expect you to write out longer proofs, do they refer to concepts not fully explored in the lectures? You might also ask your teacher for ideas on how you can improve you grades. And get hold of past years questions for the exams and course work and do them.
Secondly higher level math course often depend on more concepts from other (lower level) courses. So if you come into a lower level course you can pick it up from scratch on its own but with higher level courses you have to have a good understanding of the concepts in all of its dependent courses. For example if you are taking a course in Algebraic Geometry you will need to be familiar with topology, analysis, algebra etc. So check the prerequisites for the higher level courses that you are not doing so well at and check that you have done and understood all of them.
On books I find reading a book that gives motivation and historical background at the same time as the set textbook can help a lot in getting the big picture. And this gives a structure to hang the details of definitions, theorems and proofs from your main textbook. For example I found RBJT Allenby's "Rings, Fields and Groups" much more motivating than Birkoff and MacLane's "Algebra" because Allenby gives motivation for all the abstract definitions with examples and the historical development of the subject.
On study habits I find it hard to take in new math ideas if I am not getting enough sleep or eating well. And for me there is a definite time limit of the amount of studying I can do effectively before I need a break. Explaining a new concept or proof to someone else (either real or imaginary) "over the phone" ie verbally can be a good way to see if you really understand something.