Is there any way to characterize all different possible types of operation which are not associative?
This is too broad and subjective to answer I think. What exactly is a "type" of operation? I assume you're already talking about binary operations, so presumably a "type" of operation is one which satisfies certain identities, like the associative identity. Certain specific examples come to mind:
- Jacobi identity for Lie algberas,
- Jordan identity for Jordan algebras,
- Moufang identites for loops,
- Self-distributive laws for racks and quandles,
and certainly others (I am no an expert in nonassociative algebra). Many of the identities above are not three-variable identities, but still. Generally, interesting algebras and their identities are not chosen randomly but rather follow from certain canonical examples whose properties are generalized. The algebras are meant to represent certain structures, and the identities ensure that. For instance, Lie algberas linearize Lie groups, and similarly Jordan algebras linearize projective spaces, Moufang identities generalize octonions' alternativity, racks and quandles represent how groups act on themselves by conjugation, etc.
Ultimately there is a "type" of operation for every possible set of "words" you can pick from the free magma (or if you allow addition, free nonassociative algebra) on so many generators. (There is going to be redundancy in this - different sets of words can yield the same class of algebras.)
Can we say that if ∗ is not associative, then it must instead satisfy one of set of other possible properties, depending on any other additional operations that we have on our set $X$?
Probably not. For instance the free nonassociative algebra on some generating set strikes me as a candidate for not having any "properties" (i.e. identities).
Is it ever useful to consider an 'associative commutator' for a given non-associative ∗?
Yes. The associator is useful for instance in (efficiently) proving the octonions are an alternative algebra (which is like halfway to being associative), which is in turn useful for many things like simplifying octonion expressions and classifying subalgebras and reasoning about automorphisms of $\mathbb{O}$. The octonion associator also gives rise to the exceptional ternary 8D cross product.
There's probably a lot more you can do with it in general nonassociative algebras but I wouldn't know.
Why do Lie algebras use this Jacobi identity
Consider where Lie algebras come from. Start with a Lie group $G$. The tangent space $\mathfrak{g}$ tells you all the directions that one-parameter subgroups can point in. The addition operation on $\mathfrak{g}$ corresponds to the group operation on $G$. Indeed, the exponential $\exp:\mathfrak{g}\to G$ is approximately linear in a neighborhood of $0$ with quadratic error term. As $G$ acts on itself by conjugation (and there are many sources listing example after example to show conjugation in a group is very important), so too it acts on $\mathfrak{g}$ by conjugation. Define $\mathrm{Ad}_A(X)=AYA^{-1}$ for $A\in G,Y\in\mathfrak{g}$. If we differentiate this at $A=I$ with tangent vector $X$ we get $\mathrm{ad}_X(Y)=XY-YX=[X,Y]$, the "commutator bracket." Note the adjoint action preserves this operation, and if we differentiate $\mathrm{Ad}_A[Y,Z]=[\mathrm{Ad}_AY,\mathrm{Ad}_AZ]$ at $A=I$ again with the product rule we get the identity $\mathrm{ad}_X[Y,Z]=[\mathrm{ad}_XY,Z]+[Y,\mathrm{ad}_XZ]$, which says $\mathrm{ad}_X$ is a "derivation" (i.e. satisfies the "product rule" like a derivative, but with the commutator bracket instead of multiplication). This identity may be rearranged to the more cyclically symmetric form you know as the Jordan identity.
All of the other identities I listed above have similar stories of where they come from. The Jordan identity comes from an algebraic investigation of spaces of Hermitian matrices (which are the span of projection operators, which correspond to points in projective spaces). Apparently the Jordan identity also has an interpretation in terms of the inversion symmetry of a Riemannian symmetric space, but I don't know how that story goes. The Moufang identity comes from investigating real normed division algebras, which leads to the octonions, which leads to the alternative identities, and then the simplest four-term identities one can check are where one term is repeated. The self-distributive law for racks and quandles comes from the fact conjugation is an automorphism in a group.