Why are regular languages called "regular"? Why regular languages are called "regular"? Are there any mathematical (formal or not) characterization of that word per se?
The word is overused in mathematics in unsystematisable manner so we restrict context by formal languages.
 A: I think that ultimately we can translate regular with simple, that is to say, it needs fewer data to be defined. A regular pentagon is completely defined by giving just the length of one side, while a non-regular one may need up to five sides plus some angles.
As per formal languages, these are generated by grammars, and in Chomsky hierarchy, regular grammars are definitely the simplest, because terminals and non-terminals are forced to be grouped on their own side in the production rules.
A: I've never heard any reason why it is "regular", besides the fact that a regular expression is one that can be recognised by a finite state machine, which in turn is realizable as a fairly easy electrical circuit. So, my guess is that people named the kind of language that was easiest to handle and to understand "regular". 
This is similar to "complex" numbers, which got their name at a time where most people were very puzzled about this new structure.
BTW: In the beginning, people construction computers were electrical engineers. I know that the data structure "stack" got its German name "Keller" = cellar because the circuit diagram looked like stairs leading downwards to the people who invented it.
