Non-associative commutative binary operation Is there an example of a non-associative, commutative binary operation?
What about a non-associative, commutative binary operation with identity and inverses?
The only example of a non-associative binary operation I have in mind is the commutator/Lie bracket. But it is not commutative!
 A: Convolution of distributions is commutative, but not necessarily associative.
You have the identity $R*(S*T)=(R*S)*T$ in the case where at least two of these distributions are of compact support. If this condition is violated, there is a standard counterexample $$(1*\delta_0')*H = 1'*H=0*H=0,\quad 1*(\delta'_0*H)=1*H'=1*\delta_0=1.$$
A: Here is an example with identity element and inverses.  On $\mathbb{R}_{\geq 0}$, define $a*b = \left| a-b \right|$.  Then $*$ is clearly commutative, $0$ is its identity and the inverse of any $a$ is itself.  Yet, it is not associative, since for instance $2*(1*1) = 2$ while $(2*1)*1 = 0$.
A: Consider the class of non-associative Jacobi-Jordan algebras. They are in general not associative, for dimension $n\ge 5$, but they are always commutative Jordan algebras, hence commutative. Proposition $4.1$ gives examples of such algebras:
Proposition:
Any Jacobi-Jordan algebra of dimension $5$ over an algebraically closed field of characteristic
different from $2$ and $3$ is associative or isomorphic to the following algebra, given by
the products of the basis $\{e_1,\ldots ,e_5 \}$
as follows:
\begin{align*}
e_1\cdot e_1 & = e_2,\; e_1\cdot e_4=e_4\cdot e_1=e_5,\\
e_1\cdot e_5 & = e_5\cdot e_1 =-\frac{1}{2}e_3,\; e_2\cdot e_4=e_4\cdot e_2=e_3.
\end{align*}
This algebra is not associative. It is, up to isomorphism, the unique nilpotent Jordan algebra
$A$ of nilindex $3$ with $A^3\neq 0$.
A: I asked the SMT solver Z3 to provide a function that is not associative but commutative. The way to do this is to assert commutativity for all possible inputs and ask for a counter-example for the associativity property.
The Z3 script in SMTlib format is:
(declare-fun f (Int Int) Int) ;We are looking for a function f

(assert (forall ((a Int) (b Int)) (= (f a b) (f b a)))) ;That's commutative

;With one non-associative counter-example
(declare-fun a () Int)
(declare-fun b () Int)
(declare-fun c () Int)
(assert (not (= (f a (f b c)) (f (f a b) c))))

(check-sat)
(get-model) ;Find the function

And the resulting model is:
(model 
  (define-fun b () Int ;Counter-example
    1)
  (define-fun a () Int ;Counter-example
    0)
  (define-fun c () Int ;Counter-example
    2)
  (define-fun f!1 ((x!1 Int) (x!2 Int)) Int ;Sub-function
    (ite (and (= x!1 0) (= x!2 3)) 4
    (ite (and (= x!1 0) (= x!2 1)) 5
    (ite (and (= x!1 5) (= x!2 2)) 6
    (ite (and (= x!1 3) (= x!2 0)) 4
    (ite (and (= x!1 1) (= x!2 0)) 5
    (ite (and (= x!1 2) (= x!2 5)) 6
      3)))))))
  (define-fun k!0 ((x!1 Int)) Int ;Sub-function
    (ite (= x!1 0) 0
    (ite (= x!1 2) 2
    (ite (= x!1 3) 3
    (ite (= x!1 5) 5
      1)))))
  (define-fun f ((x!1 Int) (x!2 Int)) Int ;Resulting function
    (f!1 (k!0 x!1) (k!0 x!2)))
)

define-fun f marks the generated function with the desired properties. If you sift through its definition you can see that it is defined for a few points only. On almost all inputs the function returns 3 but it has a few special points that give the resulting properties.
define-fun a/b/c marks the counter-example to associativity.
ite means if-then-else. If argument 1 is true, then return argument 2. Else argument 3.
The function works over the integers but any other domain with enough values works as well. There are no algebraic operations being performed.
I lack the syntax skills to put the function into a nice format that is customary on this site. I can translate to C, however:
int k(int x) { return
 x == 0 ? 0 :
 x == 2 ? 2 :
 x == 3 ? 3 :
 x == 5 ? 5 :
 1;
}

int f1(int x, int y) { return
 x == 0 && y == 3 ? 4 :
 x == 0 && y == 1 ? 5 :
 x == 5 && y == 2 ? 6 :
 x == 3 && y == 0 ? 4 :
 x == 1 && y == 0 ? 5 :
 x == 2 && y == 5 ? 6 :
 3;
}

int f(int x, int y) {
 return f1(k(x), k(y));
}

Mostly, tables of inputs and outputs with a "catch-all" case for all otherwise unhandled cases.
You can execute the code and play with it online.
I thought this would be an interesting example of such a function because it is so simple. It's only structure is basically a point-wise definition with just a few such points. Many of the other answers have given much more complicated mathematical examples. I hope this is an interesting complement to them.
It was mentioned in the comments that IEEE floats pose another example. Z3 can reason precisely about floats. In case you're interested here's a proof that float addition is commutative: http://rise4fun.com/Z3/CplfO This could be extended to check non-associativity as well.
A: Write down a blank $n \times n$ matrix, and randomly fill in the entries on and above the diagonal with values from $\{1,\ldots,n\}$.  Then mirror these into the below-diagonal half to make it a symmetric matrix.
Reading this as a multiplication table, it determines a commutative operation on $\{1,\ldots,n\}$.  The probability of this being associative gets very small quickly as $n$ grows.  But — the precise probability aside — if you just try it a few times (even for e.g. $n=3$), you will quickly find a non-associative example.
A: Consider the "rock-paper-scissors" operation $\ast$ on the set $\{R, P, S\}$ where we declare each element to be idempotent and declare the product of two distinct elements to be the winner according to the usual rules of Rock, Paper, Scissors:
\begin{array}{r|ccc}
\ast & R & P & S\\
\hline
   R & R & P & R\\
   P & P & P & S\\
   S & R & S & S .
\end{array}
The Cayley diagram is symmetric, so $\ast$ is commutative, but $$(R \ast P) \ast S = P \ast S = S \neq R = R \ast S = R \ast (P \ast S),$$
so $\ast$ is not associative.
A: Consider $\Bbb R$ endowed with the commutative arithmetic mean operation
$$a \oplus b := \frac{a + b}{2}.$$
It is not associative, as
$$(a \oplus b) \oplus c = \frac{a + b + 2c}{4}$$ but $$a \oplus (b \oplus c) = \frac{2a + b + c}{4} .$$ (The operation $\oplus$ has no identity, and hence no inverse, but it defines a quasigroup structure on $\Bbb R$, which means that for any $a, b \in \Bbb R$ there is $z \in \Bbb R$ such that $a \oplus z = b$.)
In fact, this construction seems to work just as well if we replace $\Bbb R$ by any (unital) ring in which $2$ is invertible.
A: If $A$ is an abelian group, then the new operation $a * b = - (a+b)$ is commutative but is typically not associative.   (This comes up naturally in the classical chord-tangent law on a cubic curve $C$ in the projective plane: if $a$ and $b$ are two points on the curve, then $a*b$ is the third point of  intersection of the line through $a$ and $b$ with $c$.)
A: The simplest example of a nonassociative commutative binary operation (but lacking an identity element) is the two-element structure $\{a,b\}$ with $aa=b$ and $ab=ba=bb=a;$ note that $a=bb=(aa)b\ne a(ab)=aa=b.$ This is the NAND (or NOR) operation of propositional logic, where $a=\text{true}$ and $b=\text{false}$ (or vice versa). See this answer or this one.
A: Adding a little to @DietrichBurde's answer, Jordan algebras (characterized by $xy=yx$ and a limited associativity $(xy)(xx)=x(y(xx))$ are a reasonably natural not-fully-associative class of algebras. The standard Jordan algebras are made from associative algebras with multiplication $\cdot$ by defining $xy=x\cdot y+y\cdot x$. For non-commutative algebras, this gives a non-associative Jordan operation. For example, such notions are useful in harmonic analysis on the associated cones: for the associative algebra of $n$-by-$n$ real matrices, the associated Jordan algebra structure is useful in studying the cone of positive-definite $n$-by-$n$ matrices. Jordan's original motivation was from observables in physics, but/and, again, self-adjoint things are prominent.
