The $4\times 4\times 4$ cube and higher aren't groups in the same sense that the $3\times 3\times 3$ cube is a group.
The set of reachable positions of a $3\times 3\times 3$ cube, viewed as functions from a $54$-element set (representing the locations of the stickers) to a $6$-element set (the colors of the stickers) form a group. The operation here is given by the following: For positions $x, y$ of a $3\times 3\times 3$ cube, let $a_1a_2 \cdots a_n$ be a sequence of moves which, starting from the identity, puts the cube into position $x$, and $b_1b_2 \cdots b_m$ the same for y. The product $xy$ is the state the cube is in after the sequence $a_1 \cdots a_n b_1 \cdots b_m$.
The fact that this operation indeed forms a group isn't as trivial as it first seems. The issue isn't with associativity, identity, or invertibility. Instead, it's with well-definedness. How do you know that the choice of sequences $a_1 \cdots a_n$ and $b_1 \cdots b_m$ doesn't make a difference?
For the $3\times 3\times 3$ cube, the way to solve this is viewing it as a subgroup of the symmetric group on the set of all $54$ stickers. This doesn't work for larger cubes, because it's possible to come up with moves that move some of the cubies around, without changing how the stickers show. The stickers can move, without their apparent colors changing. To see why this is impossible for a $3\times 3\times 3$ cube, note that any cubie is uniquely specified by its stickers, so any permutation of 2 stickers of the same color in the cube group must permute the cubies, which then cannot be the identity.
But on the $4\times 4\times 4$ cube, this fails. All $4$ of the starred stickers here, for example, are indistinguishable, in the sense that you could permute them and still have a solved state. I don't think a permutation of just these $4$ stickers is possible, but there are many permutations of indistinguishable cubies which can be done. According to Dustan Levenstein in the comments, any even permutation of these 4 stickers (or more generally of any center stickers) is possible.

The way to prove formally that the $4\times 4\times 4$ cube is not a group is to find a sequence of moves that acts as the identity on one configuration, but not on another configuration. This is pretty easy to do, but I don't remember the precise solution, so I'll omit it. (If anyone really wants such an example, comment and I can dig one up.)
It is true that if we labeled all of the stickers somehow, forming a so-called "supercube", and required that the labels also match up, then we would have a group. This group would be constructed in the same way as the $3\times 3\times 3$ group, as a subgroup of the symmetric group on the $96$ stickers of the $4\times 4\times 4$ cube.
This group acts on the set of positions of the $4\times 4\times 4$ cube transitively, but not freely. This, in group theoretical language, is why the $4\times 4\times 4$ cube positions do not form a group. We can still study the larger group, but we need to take into account that the action isn't as nice as in the $3\times 3\times 3$ case.