What you're looking for is the following:
(1) You want a binary operation on $\mathbb{Z}$. That is, you want a function $f: \mathbb{Z} \times \mathbb{Z} \to \mathbb{Z}$. It's common to use multiplicative notation for any binary operation (even if the operation is not literally multiplication). Thus we could write $x * y = f(x,y)$.
(2) This operation should be idempotent: for all $a \in \mathbb{Z}$, we require $a * a = a$.
(3) This operation should be commutative: for all $a, b \in \mathbb{Z}$, we require $a *b = b*a$.
(4) This operation should satisfy $\textrm{min}\{ a, b \} < a * b < \textrm{max} \{a, b \}\ldots$except in the cases where it can't. For example, there is no integer $z$ that satisfies $6 < z < 7$, so you're forced to accept a value of either $6$ or $7$ for $(6 * 7)$. Similarly, you accept $a * a = a$ for all $a$, as in (2).
Condition (4) is a bit complicated. Things might get a little simpler if you're willing to RELAX condition (4) in favor of just always allowing non-strict inequalities. So, your question didn't ask about this, but you might be interested in what happens if we replace (4) with (4'):
(4') For all $a, b \in \mathbb{Z}$, we require $\textrm{min}\{ a, b \} \leq (a * b) \leq \textrm{max} \{a, b \}$.
Or equivalently:
(4'') If $a \leq b$, then $a \leq (a*b) \leq b$.
This allows for some new examples. To begin with, the operations $x * y = \textrm{min}(x, y)$ and $x * y = \textrm{max}(x, y)$ are obviously two functions that would satisfy properties (1-3) and (4''). Notice that $\textrm{min}$ and $\textrm{max}$ are the meet and join, respectively, of the partially ordered set $(\mathbb{Z}, \leq)$.
In general, we could replace $(\mathbb{Z}, \leq)$ with any lattice; the meet and join functions will always satisfy conditions (1-3). It should be clear that in any lattice, if $a \leq b$, then $\textrm{meet}(a, b) = a$ and $\textrm{join}(a, b) = b$. Therefore the meet and join operations in any lattice will always satisfy condition (4'') also.
http://en.wikipedia.org/wiki/Lattice_(order)
FYI, the meet and join will also always be associative. Notice that for example the arithmetic mean operation, $x*y = \frac{x+y}{2}$, does not have this property.
I understand that the $\textrm{min}$ and $\textrm{max}$ functions are not answers to your question; they do not satisfy your original condition (4). Still, I thought it was worth mentioning that $\textrm{min}$ and $\textrm{max}$ are sort of "degenerate averages", and that you can generalize this to any lattice (which includes any totally-ordered set).