The factorial has a straightforward interpretation in terms of automorphisms/permutations as the size of the set of automorphisms.
One possible generalization of matrices is the double category of spans.
So an automorphism $ R ! = R \leftrightarrow R $ over a span $A \leftarrow R \rightarrow B$ ought to be a reasonable generalization.
I usually find it easier to think in terms of profunctors or relations than spans.
The residual/internal hom of profunctors $(R/R)(a, b) = \forall x, R(x, a) \leftrightarrow R(x, b) $ is a kan extension. The kan extension of a functor with itself is the codensity monad. For profunctors and spans the automorphism ought to be a groupoid (a monad in the category of endospans equipped with inverses.)
The factorial is the size of the automorphism group of a set. The automorphism group ought to generalize to a "automorphism groupoid" of a span. I suspect permutation of a matrix ought to be a automorphism groupoid enriched in Vect but this confuses me.