Is it possible to formalize all mathematics in terms of ordinals only? Our experience shows that all finitary mathematical objects could be encoded using the natural numbers, and all operations on those objects could be expressed in terms of a few basic operations on numbers: the successor operation, addition and multiplication.
As far as I understand, this is not a theorem in the strict sense (because the notion of a finitary mathematical object is somewhat vague), but is rather a principle with a status similar to the Church–Turing thesis. Nevertheless, the essence of this principle can be expressed by a precise statement, for example, that the theory of hereditary finite sets and the Peano arithmetic are mutually interpretable.
As it turns out, other familiar operations on integers — the exponentiation, prime factorization and in fact (invoking the Church–Turing thesis), every effectively computable operation — can be expressed using the few aforementioned basic operations.
Now, the set theory encompassing infinite sets and the powerset operations is much richer than the Peano arithmetic. But we can observe that the universe of sets contains a subclass similar in some of its properties to the natural numbers — the class of ordinals. So, here is my question:

Is it possible to reduce studying of (finite and infinite) mathematical objects to studying of ordinals, and to build a formal theory free of the concept of an arbitrary set, the theory whose universe consists of ordinals only, that uses only a few basic operations on them, but the theory that is still powerful enough to serve as a foundational theory of all mathematics — in the same sense as it is possible to use the Peano arithmetic as a foundational theory for finitary mathematics? And if so, what could be its language and its axioms?

To clarify, I'm not asking if it would be useful or convenient to use such a theory, but I'm interested in a mere possibility to do that.
 A: Edit: I've just been told that Rathjen's definition of proof-theoretic ordinal is intentionally vague for the introduction and not fit to base results off of. However as far as I know one of Rathjen's more specific renderings of Gentzen's result in particular is correct, $$\textrm{PRA+}``\textrm{There exists a provably recursive well-ordering of order type }\varepsilon_0"\vdash\textrm{Con(PA)}$$
(Although another user told me this might be wrong as well, it might work if we fix an ordinal notation in advance as in (14).)

Some analysis of theories where every object in their domain of discourse is an ordinal, or "theories of ordinals" as Arai calls them, has been done. For example in the paper Proof theory for theories of ordinals II: $\Pi_3$-reflection, Arai has defined and analyzed a theory $T_3$ for universes that are $\Pi_3$-reflecting ordinals. And this theory proves the consistency of Peano arithmetic:
According to Rathjen's paper "The realm of ordinal analysis", $\textrm{PRA}$ plus transfinite induction of length $\varepsilon_0$ along a certain primitive recursive predicate is sufficient to prove $\textrm{Con(PA)}$. Since the ordinal Arai calibrated $\vert T_3\vert$ to is greater than $\varepsilon_0$, this means $T_3$ proves some primitive recursive ordering of $\omega$ with order type $\varepsilon_0$ to be a recursive well-ordering. If $T_3$ also interprets $\textrm{PRA}$, then $T_3$ proves $\textrm{PRA}+\textrm{PR-TI}(\varepsilon_0)$, and so proves $\textrm{Con(PA)}$.
