I have been working as a mathematician at a university for about ten years and now I'm earning my money as a software developer for about the same period of time, so I think I do have some experience related to this question. Nonetheless my answer is probably very subjective and I know there are reasonable and bright people out there (including professional software develpers) who will advertise a completely different view on this topic.
The short version is that the choice programming language does not really matter. If you have a compiler at your finger tips and something you want to try, just go ahead, as long as you have fun you will very likely learn something useful.
If you really want to think longer about it: there is (in my humble opinion) no programming language which is specifically suited for a particular job or profession. Specific programming languages are often well suited for specific tasks, which is quite a different thing (as a side remark: mathematicians are supposed to be able to tackle any kind of task, as soon as they leave university ;-) If you are ever assigned to a programming task you will have to use the language which is used in the project in question (usually it is not your choice) and if it actually is your choice it will depend strongly on which one is suitable for the task in question.
Learning a given programming language on the job is usually not a big issue, especially if you have a mathematical education and good abstraction and analytic skills. Writing good software which does more than calculating some numbers or drawing some nifty picuters is, independent of the programming language, quite hard though, and consists of much more aspects than just programming.
If you just want to get an idea of how programming works at all any programming language will probably do, but as a student you will very likely prefer something with a stable and free (in terms of money) development tool chain, and it will be a good idea to start out with examples which do not depend too much on a particular OS.
There are some aspect for which I would say that they might be reasons to prefer a particular language, e.g. a language with strict type checking will be good for discipline, it is easier to learn such a language first and then some which does not enforce it than the other way round.
Whether it will involve GUI programming is a matter of your personal taste as long as you have no specific task assigned to you, but in most cases this will force you into using a particular OS (unless you take Java).
Finally there is the personal preference. Some people e.g. prefer languages like Java or C# which prevent them from doing stupid mistakes (e.g. in memory management) while others prefer languages like C++ which give them full control on this kind of thing and allow them at the same time to shoot in their own foot once a day -- it is up to you what you prefer in that regard ;-)