Why is it not important what mathematical objects are? In axiomatic set theory the term "set" and the relation "$\in$" are primitive notions. Thus, it is not defined what sets are nor what the relation is. Axiomatic set theory is supposed to formalize how sets "behave", meaning what is true about them. One of the axioms is the axiom of infinity, which ensures that one can encode the natural numbers as sets, thus proving their existence from ZFC. This is one common procedure that is done in mathematics: we have an intuitive notion, namely the natural numbers "$\{1,2,3,4,...\}$" that we used for a long time and want to make this precise mathematically. Thus, when ZFC is used as the chosen foundation, one tries to encode this intuitive object using sets.
Another example where this is done is the notion of a function. Intuitively a function is an object/concept of mapping an object of one collection to exactly one object of another collection. This can again be defined in set theory: Suppose that $X$ and $Y$ are sets, then a function $f$ is defined to be a subset $f \subseteq X \times Y$ that satisfies the following condition. For every $x \in X$ there exists exactly one $y \in Y$ such that $(x,y) \in f$. Thus we have proven that there exist objects that we call "functions" that have the property we intuitively expect functions to have, or more precisely that we expect the intuitive notion of function to satisfy.
These are of course only two examples of many, where some intuitive concepts are encoded as sets. In my understanding, we have built "models" (I know this term is used with a meaning in logic, however I don't know if this is the same way its being used there; I am using it with the usual meaning of being a model for somethign) for intuitive concepts. It might be however, that, assuming the existence of mathematical objects, the actual objects are not encoded as sets, or that you are using a different definition for the same concept. In this sense, these might be the "wrong" definitions, without us knowing whether they are or not. My question is, why is this not problematic? Why do we not care what mathematical objects are but only what we can do with them/what is true about them?
An attempt of answering this myself: I suspect that this is because the models we built above satisfy the essential properties we expect the intuitive notions to have. That is, if these intutive notions actually exist, they would have to have these properties. Thus when we prove anything for our models only using these properties, they necessarily would also have to be true for the actual objects. The same goes for different definitions/models of the intended intuitive notion. They would have the same essential properties. Thus, it doesnt matter what the mathematical objects are, since our model serves to prove the properties about them, no matter what they are.
If I understand correctly, a similar view can be found in these notes that I randomly found some time ago. On page 25 of the document it says:
"The theorems that we proved for a Peano system also turn out to be true when they are
interpreted as being statements about whole numbers. This is just what one would
expect: if the axioms of $\mathcal{P}$ are true when interpreted as statements about whole numbers, then all the logical consequences of those axioms (theorems) will also be true about
whole numbers."
Another related question: When axiomatic set theory doesn't say what sets are, then could they be anything that satisfies the axioms? One intuitively has collections in mind when thinking about sets, which should be fine, as long as one uses the axioms to construct new collections. However, axiomatic set theory doesn't tell us that sets are collections. They could thus be anyhting. If this were correct, I guess one could use the same approach as above and view sets as a model for collections, meaning that the intuitive notion of a collection obeys these axioms, which means that anything we prove about sets is also true for the intuitive notion of a collection. So are sets necessarily collections?
All of this would also make it a bit more intuitive why abstract concepts such as numbers of pure math actually work when talking counting real world things, or applying these concepts in general to the real world.
 A: Here are my two cents.

My question is, why is this not problematic? Why do we not care what mathematical objects are but only what we can do with them/what is true about them?

Here's a problem: what do you mean by "what they are"? How do you answer such question? And how is that different from what maths does: describing things by axioms, by properties?
For example: what is a car? You describe it by certain properties: it has wheels, it has engine, it carries people. But what is engine? It is a device that converts energy into mechanical energy. But what is energy? Ahh, here we have an issue. It is this thing, a property, that is somehow related to work. We really don't know what that thing is, we only know how it behaves.
You see, at the end of the day there is no "what that thing is" but only "what properties this thing has".
And sure, maybe $\{0,1\}$ has some properties that $\{1,2\}$ doesn't have. But from the set theoretic point of view? Not really. To distinguish them we can apply some other theory, maybe number theory. It only depends on what my/your goal is.

When axiomatic set theory doesn't say what sets are, then could they be anything that satisfies the axioms?

Yes, that's exactly the point.

So are sets necessarily collections?

And what is a collection? The problem is that when you try to rigorously define what a collection is, you end up with something similar to axiomatic set theory. You just asked: are sets necessarily sets? Well... sure.
Or maybe you define collections differently? Now it is an interesting question. But you have to provide the definition first.
A: Mathematical vs Physical ideas:
For a physical object/ concept, a description / definition of it is a reduction from it i.e: something which may have less properties than the object. This is because the accuracy of our description is limited by the accuracy of senses we have to perceive it.  For a mathematical object/ concept, the description/ definition is the object in itself.
For example, when we speak of sets or functions, there is really no object or idea in the world which we are basing referencing the idea with. It maybe that a physical idea inspired it, but the mathematical idea is totally disjoint from any physical motivation it has.
Problems which may arise when going back and forth:
It can be found that many inconsistencies arise when we go from idea in a context to a mathematical idea then try to apply the mathematical idea in a different context of an idea similar to the original. See here.
Are mathematical definitions objective?
I believe even mathematical objects are not free from the limitations of the human mind. For example, how do we know that what you understand from reading a mathematical definition and what I understand from reading that same definition is the same? For discussion on how this issue can be resolved, see here
How does one transition from intuition to rigor?
I found the most insightful section to be in the initial chapter of Terrance Tao's Analysis where he builds up the Natural numbers. Essentially he begins with some basic objects, and some 'reasonable' rules between the objects so that the objects are actually expressive of the conceptual idea of Natural numbers.
It turns out later that we need more axioms to restrain what the objects generated out of the above premises can be, and hence axioms are added till finally the natural numbers defined through this axiomatic way is exactly the type of thing is what we want them to be.
A: This question is a bit old at this point, but I'll take a crack at it. (comments/corrections from those more knowledgable than me are welcome):
Regarding "what mathematical objects are", I'll focus on the simple case of natural numbers. My understanding of the Peano axiomatic formulation of natural numbers is that the whole point is to strip away anything incidental or inessential and to describe just the core logical essence of counting things. So as you alluded to in your hypothesized answer, the whole point (I think) is specifically to NOT prescribe any particular "realization" of the axioms (such the finite von Neuman ordinals in ZFC). As far as Peano arithmetic is concerned, each number is a totally abstract "thing" (linked one to the next by the successor relation, which is defined purely by logical statements), and that's all you really need to say about them. Furthermore any construct that obeys the Peano axioms can be considered "natural numbers" (whether that construct be defined within ZFC, some other set theory, or even a non-set-theoretic system...category theory maybe?).
From that point of view, asking which specific construct is the "real" or "actual" natural numbers is not really an applicable question. In fact, I can think of a pretty specific argument that seems like a good reason not to anoint one particular realization as the "real" natural numbers: If within set theory you define the natural numbers in the most common way, as the set of finite von Neumann ordinals (for which any ordinal is the set of all smaller ordinals), then it is true that, say, $1 \in 3$. But in Moschovakis' Notes on Set Theory he uses an alternative definition of natural numbers as
$0 = \emptyset$, $1 =\{\emptyset\}$, $2 = \{\{\emptyset\}\}$, $3 = \{\{\{\emptyset\}\}\}$, ...
in which case it is not true that $1 \in 3$. So if you were to anoint the finite von Neumann ordinals as the "true" natural numbers, then you would have to conclude $1 \in 3$ as an unequivocally true mathematical statement, which is obviously wrong (or at least highly dubious)
An analogy with the game of chess might illustrate this further. What is the "white queen" in chess? I could try to define this by, for example, describing the shape of the queen piece in a typical chess set, but that would be a mistake. Maybe someone has a chess set where the pieces don't have the conventional shapes (maybe they are all discs differentiated by logos). Maybe I'm playing computer chess, where physical pieces don't exist at all (or "blindfolded" chess, where there is no board and players just memorize the state of the game in their minds and call out moves to each other). So the logical essence of what the "white queen" is, is the following:

*

*It is an abstract "thing" (maybe a physical piece, maybe just a concept in my mind) that has a "position" property consisting of a letter from "a" to "h" and an integer from 1 to 8.

*Its initial position is "d1".

*It obeys a certain list of rules about how its position is allowed to change from one turn to the next.

That's it. That's what a white queen is. Any attempt to get more specific about what it "really" is will only lead to mistaken specificity (if I say it has a white color, that will be wrong in the case of "blindfolded" chess).
As for whether sets are containers: In pure axiomatic set theory (without urelements), it is true that for any set $X$ (other than $\emptyset$), there are some other sets that we call its "members", i.e. some other sets related to it via the membership relation. Does that mean that $X$ is a "container" that "contains" its members? That's one possible metaphor for the membership relation, but certainly not the only one. On social media apps, a given user can "follow" other users, so instead of saying $X$ metaphorically "contains" its members, you could equally well say $X$ "follows" its members. My preferred metaphor actually is to think of sets as "nodes", and the membership relation as defining "connections" between nodes. But ultimately, the notion of what a set "actually is" can't really be defined (and needn't really be defined) any more than you can define what, say, a "point" in a game of bridge "actually is". Each set in a set universe is an abstract "thing" that is related to some other sets via the membership relation, and the membership relation is defined in terms of pure logic (and not by any particular metaphor like "containing" or "following").
One final conceptual point: Your question closes by asking about "why abstract concepts such as numbers of pure math actually work when talking counting real world things". I think the whole point of the formal axiomatization of arithmetic is to replace logically imprecise "intuition" (how do I know my intuition is the same as your intuition?) with a logically rigorous formal system. If you do it right, what you end up with a formal system that matches most (if not all) people's prior intuition, so it follows essentially by definition that the formal system "works" for counting real world things. The formal system at that point becomes the "true" definition of arithmetic. If you were to find a conflict between formal Peano arithmetic and your "intuitive" calculation, there's a very small chance that you found a flaw in Peano arithmetic, but an overwhelmingly larger chance that you need to re-examine your intuition.
It might seem odd to mistrust intuition about something as basic as arithmetic, but my understanding of the reasoning behind this goes like this: In the late 1800's, mathematicians became acutely aware of the flaws in classical geometric reasoning (such as in Euclid's Elements) that makes explicit or implicit appeals to "intuitively self-evident" attributes of physical space. They correctly anticipated that such intuition might not be reliable (as Einstein later demonstrated with general relativity and curved spacetime). So work began for a logically rigorous axiomatization of geometry. At the same time, they realized that a similar level of rigor ought to be applied to subjects like arithmetic and analysis. "Intuitive" arithmetic might seem less suspect than "intuitive" geometry, but once you start reading the work of people like Dedekind, Frege, Whitehead & Russell, etc., you start to realize that maybe your intuition isn't quite as "obvious" as you thought.
A: Im not sure if the following fit what the OP is asking, anyway I will try to answer it. Any comment will be welcome!

My question is, why is this not problematic? Why do we not care what mathematical objects are but only what we can do with them/what is true about them?

This is the whole point of formalization of a subject using logic!!! Formalization means that we only cares about forms (syntactic relations between objects) as far as we knows that the axiomatic formal theory is sound and a model exists for this formal theory.
About soundness: soundness means that using axioms and some defined rules of inference any derivation using these gives true statements. In symbols it means that
$$
\text{ If }\Sigma \vdash \varphi \text{ holds then }\Sigma \vDash \varphi \text{ also holds }
$$
In other words: if there is a derivation of $\varphi $ from a set of premises $\Sigma $ using the given axioms and rules of inference then when $\Sigma $ is true (every premise is true) then $\varphi $ is also true.
About the existence of a model for a formal theory: a model is, roughly speaking, a correspondence between "real things" and a formal theory. Given a model, for every formula in a formal system we can see it as a formula of "a real thing", so as far as a model exists for a formal theory then the formal theory its useful: it reduces the complexity of the "reality" of the model to just syntactic relations.
So we don't need anymore to care about the "reality" or "existence" of some object defined by a formula in a formal theory as far as we have soundness and a model for it. In this context "reality" means some intuitive mathematical objects that we used from many centuries ago and they doesn't show to be problematic. Then the formal theory "fits" to our "real" mathematical objects.
Take a look here.
