There absolutely are non-associative structures! Relevant terms here include "loop" and "Lie algebra." For concrete examples, consider octonion multiplication or the "midpoint" operation on points in $\mathbb{R}^n$ (note that the latter is commutative but not associative!). There are also weakenings of the associative law, like power associativity or alternativity. The study of nonassociative structures is very much a thing.
However, there is definitely an important sense in which associativity is special, namely that it lines up with function composition (note that function composition is not commutative: $(x+1)^2\not=x^2+1$ in general). This is important since we generally run into groups not "in a vacuum" but rather as a component of a richer object - namely a group action. A group action is basically a way of representing elements of a group $G$ as "structure-preserving" maps on some other structure $A$. The point is that this is generally the way groups emerge in the first place: consider for example Galois theory, where we're not interested in $Gal(K/F)$ on its own so much as we're interested in the action of $Gal(K/F)$ on $K$.
And the other direction is important too: given a group $G$ we can often gain a better understanding of $G$ by thinking about its possible actions. An early example of this is Cayley's theorem, where we look at a simple action of a group on itself, and representation theory is built around the idea that we can learn a lot about a group by looking at the various ways it can act on vector spaces.
The idea of an action - where elements of the acting structure represent functions on the acted-on structure, and the operation of the acting structure represents composition - automatically enforces associativity since function composition is associative. Conversely, if we try to whip up a notion of "action" which does not automatically enforce associativity - that is, a notion of a non-associative magma acting on a structure $X$ in such a way that $[a*(b*c)]x\not=[(a*b)*c]x$ in general - things get messy. This doesn't make non-associative structures uninteresting, but it does mean that the natural idea of "action" is really only appropriate to associative structures, and this does wind up being a big deal.
