Why should the relative size of two sets depend on which other sets exist? Cantor convincingly argued for the proposition that two sets are of the same size exactly when there is some bijective function from one to the other.
However, "functions" are formally defined as sets themselves, and so the relative size of two sets can depend on which other sets (namely, functions between them) exist.
It seems to me that defining a function as a set of ordered pairs (or perhaps actually a triple- the domain, codomain, and then a set of ordered pairs) is a "neat formal trick" as Omar Antolín-Camarena says here, but "not really how anyone intuitively thinks about a function." Cantor's argument relating size to bijections depends crucially on our intuition.
Is there any way to define functions other than as sets and get around this 'issue', or alternatively formalize an absolute notion of size (something like two sets are equinumerous if a bijective function between them exists in any model of set theory)?
 A: The generalization you suggest cannot work. Given two infinite sets we can add by forcing a bijection between them. This will result in all infinities collapsing into the same cardinality, rendering this notion completely useless.
Moreover there are models which have non-standard integers. In fact, one can even have more integers in one model than real numbers in another model. So it becomes impossible to even say what is the cardinality of "the integers" since those are not absolute between models of set theory.
But let me offer you an intuitive solution as to why this phenomenon is actually common in mathematics, only when we think about sets it becomes troubling to us. I will also explain why.
How many cubic roots are there to $2$? Well, that depends on the mathematical structure we are considering. In the rational numbers the number $2$ doesn't have any number $x$ such that $x^3=2$. On the other hand, in $\Bbb R$ there is exactly one $x$ with this property, whereas in the complex numbers there are three such numbers.
So the question becomes dependent on the context in which we are working, and which numbers exist in the "universe of numbers" we might consider at the moment.
But we never really find this issue troubling. Why? Well, we are used to think about the complex numbers as absolute, and the real numbers are absolute, and the rationals are absolute, and so on. This means that there is some "large" universe of "all numbers" (which is how a non-mathematician person usually see this) where everything happens. In fact, asking people what is a number often you will get answers ranging from "Uhm, $0,1,2\ldots$" to "Something signifying physical quantity" to "A complex number".
So we have these canonical models for all the things we think of as numbers. And so there is no issue in stepping up to a larger context when needed. On the other hand set theory has no such canonical model, so we are left with some sort of uncertainty with what sort of objects live in the model we consider at the moment.
The fact that we can switch models causes a lot of tension to people first hearing about it. How can sets be different between models? Well, how can $x^{412341}-2$ have so many different solutions in different contexts? Because context matters.
Equinumerosity depends on the context, much like the other question. It's fine.
A: Something along the lines of what you are proposing was developed in the article
Benci, Vieri; Di Nasso, Mauro, Numerosities of labelled sets: a new way of counting. Adv. Math.  173  (2003),  no. 1, 50–67.
(the notion of numerosity is more "flexible" than that of cardinality).
Jörg D. Brendle's review of
Blass, Andreas; Di Nasso, Mauro; Forti, Marco, Quasi-selective ultrafilters and asymptotic numerosities. Adv. Math.  231  (2012),  no. 3-4, 1462–1486
here specifically compares numerosities and cardinalities.
