Are there rules in the useage of prepositions in Math? It is often to use prepositions in various expressions. 
E.g.


*

*$2$ is in the set of natural numbers $\mathbb N$ 

*The symmetric group on 3 letters $S_3$ is the group consisting of all possible orderings of the three letters ABC i.e. contains the elements ABC, ACB, ..., up to CBA, in total 6 elements. 

*The group of integers modulo $n$ is written $Z_n$ or $Z/nZ$

*If $R$ is a binary relation over $X$ and $Y$, then $R^{-1}$ is a binary relation over $Y$ and $X$.
However, it's perhaps a bit dazzling for someone who is not quite familiar to the syntax of English. So my question: Are there some rules in the useage of prepositions such as 'at', 'in', 'on', 'over', 'under', 'of', 'for' and 'from' in Math? Or everything is just a convention?
 A: Beyond all of these prepositional stuff that you mentioned, every word in a daily language to express a mathematical statement is just used as a convention or somewhat. So, within a mathematical view which needs rigour essentially, you often lose precision this way when you try to put a mathematical statement. To say in short, daily language is not a powerful tool to express mathematicians "words". In order to deal with this problem as much as possible, mathematicians use formal logic where there are just symbols whose meaning is already known by definition or a priori.
A: I think your usage is clearly understood, and in the middle two propositions, you define what it is you're saying, so it's very clear. 
I agree with Chris that the usage of "binary relation on" is much more common that "over", and in any case, you could easily include: 


*

*If $R: X \times Y \to Z$ is a binary relation on $X$ and $Y$, then $R^{-1}: Y \times X \to Z$ is a binary relation on $Y$ and $X$.


Likewise, you could easily write 


*

*"$2$ is in the set of natural numbers $\mathbb N \iff 2 \in \mathbb N$."


Then, in all cases you are essentially defining what your expressions mean.

The words you mention are, of course, words often preferred by convention only. Readability and rigor are often at odds with each other. When writing expository mathematics, we rarely see proofs in which everything is written in formal logic, and to write in such a way for most journals, certainly in textbooks or theses, would be rather awkward. So we sacrifice precision, when appropriate, for the sake of readability. But for introducing definitions and preceding crucial proofs, we need to make our language unambiguous by more precise, formal means.
On this note, you might find the entry on Terence Tao's blog helpful: 


*

*Advice on writing papers: On Writing.

A: If we're discussing grammar, I am not sure what the difference between "rule" and "convention" is, and I suggest that you're best advised to follow the conventions used by competent writers. Responding to your actual question, you will find discussions on the use of prepositions on the web, I'm sure. But if English is not your mother tongue then I suspect it's best just to learn the combinations on a case-by-case basis. 
Also, I agree with you usage in the first three examples but I think "binary relation on" is much more common.
A: The short answer is that you should speak and write the way that a native speaker of the language does. If you’re uncertain, ask a native speaker. Note that this is true for all languages, not just English.
The long answer is that English is one of the Indo-European languages, which all (as far as I know) put a very great weight on the preposition. And the wrong preposition can change the meaning of a sentence disastrously, or merely make the sentence sound clumsy. Furthermore, which preposition is the right one can be entirely different not merely from language to language (in English, $x^3$ is “$x$ to the third”, but in Danish it’s “$x$ in third”) but from dialect to dialect within a language (among Americans, $x/y$ ix “$x$ over $y$”, but among Britons it’s “$x$ on $y$”, as I recall). In short, then, see the paragraph preceding.
