Is there a common theme between modulus, moduli and modulo? The moduli space is some sort of space of parameterizing space, modulo as in modular arithmetic and modulus as in the modulus of a complex number.
Is there a reason all these words are the so similar?
 A: They all have the same root. "Modulus" means "measure"; a modulus is a measure of something. The modulus of a complex number measures its length as a vector. The term "modular arithmetic" comes from Gauss who used "modulus" as follows:

Si numerus a numerorum b, c differentiam metitur, b et c secundum a congrui dicuntur, sin minus, incongrui; ipsum a modulum appelamus. Uterque numerorum b, c priori in casu alterius residuum, in posteriori vero nonresiduum vocatur. [If a number $a$ measure the difference [emphasis mine] between two numbers $b$ and $c$, $b$ and $c$ are said to be congruent with respect to $a$, if not, incongruent; $a$ is called the modulus [emphasis mine], and each of the numbers $b$ and $c$ the residue of the other in the first case, the non-residue in the latter case.]

So in modular arithmetic $n$ is the modulus and working $\bmod n$ means measuring the difference between two integers in units of $n$. Remember that the Western mathematical tradition is deeply steeped in Euclidean geometry and for over a thousand years a number was a length. "Measuring the difference" means that $n$ divides the difference but the term "measuring" is meant (I believe, anyway) to evoke the geometric picture of a stick of length $n$ being used to literally measure the length of the difference.
"Moduli" is the plural of modulus so "moduli space" should be interpreted as "space of moduli," in the sense that a moduli space describes all possible ways an object of some type can vary (and we can measure these variations).
