How are groupoids richer structures than sets of groups? This has been bugging me for quite some time: My intuition with categories is, that I can simply identify isomorphic objects. It does for example not matter, whether the entries in a sudoku are the numbers $1,2,\dots,9$ or letters $a,b,\dots,i$ (this shows, that you can simply identify isomorphic sets).
I heard groupoids are important objects, possibly even more fundamental then categories (I even dealt with them before). But this seems to contradict my intuition, for you could mentally identify isomorphic objects in a groupoid and end up with just a set of groups. There has to be something wrong with this view and I suppose it has to do with the fact, that there are usually many isomorphisms between isomorphic objects (It is well known from linear algebra, that choices of bases "matter"). I realize that this is a very imprecise question, but:

How can I think about groupoids, such that there are more interesting
  or richer in structure than just sets of groups?

 A: The first short answer is that in order to identify a groupoid with a set of groups you need to pick a basepoint in each connected component (in more categorical terms, a representative of each isomorphism class), and there are various situations where you don't want to (analogous to why you often don't want to pick bases of vector spaces).  
The second short answer is that there are many reasons to consider groupoids with extra structure, which can be considerably more interesting than sets of groups with extra structure.
Here is an example where both of these considerations apply. Suppose a group $G$ acts on a space $X$. Does this induce an action on the fundamental group? The answer is no: in order to get such an action, $G$ must fix a basepoint of $X$. But it can happen that $G$ fixes no basepoint (even in a homotopical sense). However, $G$ will always act on the fundamental groupoid of $X$.
For example, let $X$ be the configuration space of $n$ ordered points in $\mathbb{R}^2$. This space has fundamental group the pure braid group $P_n$, which fits into a short exact sequence
$$1 \to P_n \to B_n \to S_n \to 1$$
where $B_n$, the braid group, is the fundamental group of the configuration space of $n$ unordered points. Now, it's clear that $S_n$ acts on $X$ by permuting points. But this action cannot be upgraded to an action on $P_n$, because the above short exact sequence does not split. 
This is not an isolated example. It's part of the reason why the $E_2$ operad can be described as an operad in groupoids, but not as an operad in groups, even though its underlying spaces (homotopy equivalent to the configuration spaces above) are all Eilenberg-MacLane spaces. 
There are lots of other things to say here. For example, groupoids form a 2-category, groupoids are cartesian closed, topological groupoids are richer than topological groups... the list goes on and on. Here is a slightly cryptic slogan: 

You cannot really identify isomorphic objects. The space of objects isomorphic to a fixed object $X$ is not a point, it is the classifying space $B \text{Aut}(X)$. 

