The name refers to the way we choose to group our items. The way we've evolved, we found that nine counting symbols and a symbol for nothing suffice for our preferred base. We're able to recycle the glyphs $1$ and $0$ to denote our grouping, which is given by the combined symbol $10$.
To illustrate why it's all about the grouping, and not about the final unique symbol, let's roll with this example. Pick an amount of symbols that can use to enumerate your fingers. You may prefer $$1,2,3,4,5,6,7,8,9,X$$ but another could have just as easily chosen $\odot,\triangle,\square,\cup,\star,\vee,\uplus,\times,+,\LaTeX$. These are just glyphs at this point, markers for each of the fingers. For convenience, let's use my suggested set.
We have an issue, I want to add more symbols to keep track of toes. Okay, how about:
Man, that's a lot of symbols. I like the number of my fingers, $X$, and in fact, you could say I have $X$ toes. Wow, grouping things by $X$ is convenient, both physically and visually. Can I keep track of these groups by repeating my symbols to enumerate larger quantities in groups of $X$?
Well, it would be nice if I started counting from nothing, so let's add a symbol for that.
Now, how should this work? Well, I can have $X$ objects. Or I can have $X$ groups of $X$ objects each. Or, wow, even $X$ groups of $X$ groups of $X$ objects! It seems like reaching $X$ marks a sort of restart point. I could denote this regrouping by saying I have $1$ $X$ and $0$ other stuff. Okay, I feel inspired. Let's combine those symbols to replace $X$:
And on and on it goes... This was more or less a colorful way of expressing BigBearzzz's answer: each digit's location denotes a higher grouping. We don't group every nine objects, though. We group ten objects.
What about other bases? Well, say we choose to group every three objects. We would still want a symbol for nothing, $0$, another for one $1$, for two $2$, and once again we can recycle our glyphs to call one group of three $10$. That's where we regroup. Note that these are just glyphs, recycled from the base ten representations. The meaning is different here than in base ten due to where I choose to regroup. Why is this called base three? Well, I'd like to refer to this new system, and I choose to borrow the name from a convenient source: my favorite base of all, ten.