Velleman could be read as saying that (i) in English, "or" has two meanings when used as a propositional connective, one corresponding the familiar inclusive disjunction of formal logic, the other expressing exclusive disjunction, but (ii) there's a special convention that in mathematical English, "or" is used only in the first way. Hence the OP's question -- why the special convention in maths? Two points about this.
(I) Arguably (i) is just wrong. As background recall that a standard approach to explicating how we manage to interpret what we read or hear is that the overall message conveyed is a result of the interaction of two things, first the semantic content (the "literal meaning") of the sentence used, and second contextual and pragmatic clues.
It is not infrequently said that "or" is semantically ambiguous, i.e. it has two different literal meanings. But arguably a much smoother theory seems to be that "or" is in fact semantically unambiguous, meaning inclusive disjunction. And on those occasions where we hear/read an utterance of "$A$ or $B$" as also conveying "but not both", the additional implicature is either deduced from background knowledge that $A$ excludes $B$ or has some other contextual, pragmatic, source. (Sometimes the context is unclear or complicated and we don't know whether the speaker does or does not mean to rule out the case where both disjuncts are true: but that doesn't mean that the meaning of the sentence used is not determinate.)
Note for example that in the sort of cases typically invoked to supposedly illustrate the uses of exclusive "or", it would -- on the semantic story -- be a contradiction to add "or both" whereas it normally seems like a coherent cancelling of a pragmatic (typically Gricean) implicature. Note again that 'either ... or' in English seems to have a uniform semantic negation, 'neither ... nor ...' (which couldn't be the negation if 'or' is exclusive). And so it goes.
There is a large literature on this, unsurprisingly. For a recent review, see Lloyd Humberstone's bible, The Connectives (MIT, 2011) -- which is 1492 pages mostly on 'and', 'or', 'if' and 'not'! §6.12 is titled "Exclusive/inclusive" and gives considerations against the semantic ambiguity thesis.
(II) On a charitable reading, Velleman is probably not actually asserting (i), i.e. he is not actually defending the disputed theory that ordinary 'or' is semantically ambiguous. He indeed notes that logicians distinguish an inclusive from an exclusive formal disjunction. But he is sensibly not pausing to fuss about whether we are right to sometimes interpret bare "or" in English as semantically meaning exclusive disjunction. In maths at any rate, he is saying, we do take "or" to be by default the usual Boolean inclusive disjunction which has very nice properties like being nicely dual with conjunction, satisfying De Morgan's Laws etc., being nicely related to existential quantification, etc. etc. Those neat features of logical incisive disjunction are reason enough to concentrate on it (and we can always add a "but not both" clause if it is important to formalize an exclusive message). But note Velleman could, consistently with what he says, add the whispered aside that I would add here: "Pssst! Just between you and me, I think that's what "or" always semantically means, but it would be far too distracting to argue the case here."