What course sequence should I study in order to build a foundation for studying cryptography and signal processing? I studied mathematics about two decades ago, but unfortunately, I remember little of it. I'm hoping to start studying cryptography and signal processing, but I'm not entirely sure what course/self-study sequence to follow. My initial ideas are something along these lines:


*

*Number theory

*Linear algebra and multivariable calculus

*Abstract algebra and probably some coding theory

*Analysis, both real and complex

*...


Given how little I remember, I'm not sure how much overlap I might be missing. For example, I know that coding theory utilises quite a bit of group theory, but I don't quite know the extent of the overlap.  
 A: For cryptography, you need some math courses in number theory and abstract algebra, but it sounds like you know about those.
We have three main cartographic ciphers in the form of symmetric ciphers (algorithmic based like the Advanced Encryption Standard - AES), asymmetric ciphers and stream ciphers.
Asymmetric ciphers are based on 'trap-door' mathematical functions that use the integer factorization problem (like RSA), discrete log problem (like Diffie-Hellman) or elliptic curve discrete log problem (elliptic curve crypto).
Cryptography Books


*

*A Course in Number Theory and Cryptography, Neal Koblitz

*Algebraic Aspects of Cryptography, Neal Koblitz

*Elliptic Curves: Number Theory and Cryptography, Second Edition (Discrete Mathematics and Its Applications), Lawrence C. Washington

*The mathematics of ciphers: number theory and RSA cryptography, Coutinho S C, Severino Collier Coutinho

*An Introduction to Cryptography, Richard A. Mollin

*Introduction to Cryptography with Coding Theory (2nd Edition) by Wade Trappe and Lawrence C. Washington (Jul 25, 2005)

*Handbook of Applied Crytography, Alfred Menezes, Paul van Oorschot, Scott Vanstone

*Guide to Elliptic Curve Cryptography, Darrel Hankerson, Alfred Menezes, and Scott Vanstone

*Practical Cryptography, Niels Ferguson, Bruce Schneier
Also note that there is also the field of cryptanalysis, which is more statistically and probability theory focused.
You might want to peruse the MIT Open Courseware on Crypto for further guidance on courses and areas to brush up on.
Here are some notes and here is another set of notes (and you can find other very good mathematically based ones online too).
Here is a list of topical areas that involve crypto to get an idea of scope.
For cryptanalysis read these from an expert.
That should be enough to help guide you forward and to help you figure out what courses you need.
Regards
A: I also picked up cryptography as a hobby and took a few introductory courses and then just studied by myself. You are of course right about abstract algebra but just want to emphasize that an intro to abstract algebra won't be enough. You'll need to get a bit deeper into field theory so if somebody says to add two numbers in $\mathbb{F}_{2048}$ you would know how to do that. Finite fields (especially of characteristic two) and then finite vector spaces built on top of them are ubiquitous in cryptography. And these don't behave like real or complex vector spaces at all. You also want to pick up some probability. Of course, depending on if you want to study just theory or actual implementations and why is AES the way it is and what is differential cryptanalysis then some comp sci background will also help.
For signal processing, are talking about just fourier analysis or any thing "modern" that's out there (like wavelets)? For that real and complex analysis will help. Intro to PDEs will be useful too learning about fourier series and fourier transforms. And then the most useful thing will be a really really good numerical analysis sequence. First start with the basics of fourier transform, then the discrete fourier transform, and then understanding what is fast fourier transform. And then whatever additional techniques you want to learn like radial basis functions and/or wavelets.
