Under what circumstances is a finite group uniquely determined by its “conjugation table”?

Let $$a\mathop{.}b \stackrel{\text{def}}{=} aba^{-1}$$ denote conjugation by $$a$$

Suppose we define a matrix $$M$$, the "conjugation table", associated with our finite group $$G = (X,*_{\small{G}})$$ as follows. (I'm considering the cells of $$M$$ to be formal sums of group elements (with the product of monomials defined in terms of the group operation), but I'm only using that machinery to talk about equivalence up to relabeling.)

$$M_{ij} \stackrel{\text{def}}{=} x_i \mathop{.} x_j = x_i x_j x_i^{-1}$$

I'm also thinking of two matrices $$M$$ and $$M'$$ as equivalent if they only differ by a permutation / relabelling, so

$$M \sim M' \stackrel{\text{def}}{\iff} MP=M' \;\;\text{where P is a permutation matrix}$$

or equivalently

$$M \sim M' \stackrel{\text{def}}{\iff} M_{ij} = M'_{\sigma i \sigma j} \;\;\text{where \sigma is a permutation}$$

I can think of a case where an $$M$$ does not uniquely identify a group and a case where an $$M$$ does uniquely identify a group.

I think a group is Abelian if and only if the following holds. (The "if" direction is trivial).

$$x_i \mathop{.} x_j = x_j \;\;\forall i,j$$

So, if $$G$$ has four elements and is Abelian, then it could be the cyclic group on four elements $$Z_4$$ or the Klein four group $$V_4$$.

$$Z_4$$ and $$V_4$$ are indistinguishable by their "conjugation tables".

However, if $$G$$ has three elements, it can only be $$Z_3$$ since there's only one group of order 3.

So, there are at least some circumstances under which a given $$M$$ is associated with exactly one group. Do we know what those circumstances are?

Basically, what you call "conjugation table" is the left $$G$$-set structure on $$X$$ that is defined by conjugation. The equivalence is nothing but isomorphism as $$G$$-sets.
From the $$G$$-set structure on $$X$$, several invariants of $$G$$ can be calculated.
• the order of $$G$$ = the size of $$X$$
• the number of conjugacy classes of $$G$$ = the number of $$G$$-orbits in $$X$$
• the sizes of conjugacy classes $$\{\, \lvert g^G \rvert : g \in G \,\}$$ = the sizes of $$G$$-orbits in $$X$$ (as multisets)
• the sizes of centralizers $$\{\, \lvert C_G(g) \rvert : g \in G \,\}$$ = the sizes of stabilizers $$\{\, \lvert G_x \rvert : x \in X \,\}$$ (as multisets)
• the inner automorphism group $$\operatorname{Inn}(G)$$ = the automorphism group $$\operatorname{Aut}_G(X)$$ as $$G$$-set
So one of the simplest circumstance "when a conjugation table determines a finite group" is when $$G$$ has trivial center $$Z(G) = 1$$. This is because $$\operatorname{Inn}(G) \cong G/Z(G)$$ in general.