Inner product space over generalized number systems Apologies for the lengthy setup, but I want to make sure I am clear on how I am using the notation, and what I mean by the phrase "generalized number system".
Define a generalized number system $G$ as a set of elements equipped with


*

*addition operation, and $G$ is an abelian group under this operation (so addition is commutative and associative, and the existence and uniqueness of additive inverses also gives us subtraction)

*multiplication (and $G$ is closed under this operations)

*conjugate operation
$\overline{a}$ denotes the conjugate of $a$, with the properties that for all $a,b \in G$
$\overline{a} \in G$
$\overline{\overline{a}}=a$
$\overline{(a+b)}=\overline{a}+\overline{b}$
$\overline{(ab)}=\overline{b}\,\overline{a}$

*a linear order relation on the subset of elements for which $\overline{a}=a$  

*multiplication and addition are "compatible", via the distributive property:
$a(b+c) = ab + ac$


Note that multiplication is not necessarily commutative, nor associative, nor provides a definition for division.

Side question: Is there a more mathematically appropriate term for what I am calling a generalized number system?

Now in the special case of $G$ being some field $F$, then given a vector space $V$ over $F$, we have an inner product space if we equip the vector space with a map
$\langle \cdot, \cdot \rangle : V \times V \to F$
which has the following properties for all $x,y,z \in V$ and $a \in F$:
$\langle x,y\rangle =\overline{\langle y,x\rangle}$
$\langle ax,y\rangle = a \langle x,y\rangle$
$\langle x+y,z\rangle = \langle x,z\rangle + \langle y,z\rangle$
$\langle x,x\rangle \geq 0$
$\langle x,x\rangle = 0 \Rightarrow x = 0$.
From this, an additional property can be derived:
$$\langle x,ay\rangle = \overline{\overline{\langle x,ay\rangle}} = \overline{\langle ay,x\rangle} = \overline{a \langle y,x\rangle} = \overline{\langle y,x\rangle}\,\overline{a} = \langle x,y\rangle \overline{a}$$
using that this is a field, we can further simplify:
$$\langle x,ay\rangle = \langle x,y\rangle \overline{a} = \overline{a} \langle x,y\rangle = \langle \overline{a}\,x,y\rangle$$  
An "operator" on this vector space, will be defined as an map from the vector space back on itself
$\hat{A}: V \rightarrow V$
and the adjoint operator of $\hat{A}$ will be denoted $\hat{A}^\dagger$ is defined as the operator which satisfies: 
$\langle \hat{A}x,y\rangle=\langle x,\hat{A}^\dagger y\rangle$
This is an important setup for considering self-adjoint operators in Hilbert space for quantum mechanics.  Since people have played with quaternionic Hilbert spaces, there must be a way to generalize inner product spaces over non-commuting number systems (at least in some specific cases).
However, it is not clear to me how to generalize inner product spaces over non-commuting number systems.  Even some of the defining properties require care, for now:
choosing the requirement on the inner product to be
$\langle ax,y\rangle = a \langle x,y\rangle$
would not be equivalent to choosing it to be
$\langle ax,y\rangle = \langle x,y\rangle a$  
And furthermore, following the logic above, it does not appear to be possible to obtain:
$\langle x,ay\rangle = \langle \overline{a}\,x,y\rangle$
without commutative multiplication.  Which appears to indicate even a very simple operator defined by linearly scaling a vector by a constant
(for all $x \in V$, $\hat{A}x = sx$ where $s \in G$), does not have an adjoint operator.  So quite an amount of care must be taken when extending these concepts.

So how can one generalize inner product spaces to allow them to be over more general number spaces?

 A: (This answer makes the additional assumption that multiplication is associative in your number system. There are difficulties even with this restriction. For more generality, you might want to look at composition algebras, for example.)
The number system with addition and multiplication as you describe it is called a ring.
The conjugation you refer to is called an involution.
(You should also demand that $\bar1=1$ if $1$ is the multiplicative identity.)
Therefore your number system would generally be called an involutive ring.
(In more generality, some kind of an involutive algebra.)
If you consider something like a vector space but replace the underlying field with a ring, the thing you end up with is called a module.
In modules you have to pay attention to the direction from which you multiply my scalars.
In an inner product it might be a good idea to consider it as a pairing between a left module and its opposite module, so that for a scalar $a$ you would have $\langle ax,y\rangle=a\langle x,y\rangle$ and $\langle x,ya\rangle=\langle x,y\rangle a$.
If the module is commutative (multiplication is commutative but inverses need not exist), then things are somewhat simpler, but in general you will lose a lot of structure when going from vector spaces to modules.
There is one line in your question that I want to discuss: $\langle x,x\rangle\geq0$.
This implicitly means that there is a meaningful order in the underlying ring or field.
An ordered field necessarily has $\mathbb Q$ as a subfield, so all finite fields are excluded, for example.
The complex field is not ordered, but the inner product is defined so that $\langle x,x\rangle$ is always real, so the condition makes sense.
If you want this positivity condition for your inner product, make sure your underlying ring or field has an order.
The thing you seem to be after is something that I would call an inner product module, but I have not heard of such things being studied.
It may well be that they do not have enough structure to make an interesting theory, but I don't know.
A: Partial answer to side question:
$G$ is a nonassociative ring if we assume multiplication is also left distributive (it also does not assume the existence of 1). Also, I am reasonably sure that calling the operation conjugation implies all of the properties (before the side question) besides the linear order.
You could also call the conjugate an adjoint (adjunction?) operation on multiplication but it may be confusing since you defined another adjoint. Lastly, I am not sure what of this is standard terminology as the only source I have on hand is Wikipedia.
A: I think the answer is that unfortunately on the face of it at least, nonassociative linear algebra doesn't really make much sense. 
In the associative case, it is very simple to define a module $M$ over a ring $R$: $M$ is an abelian group and there is a bilinear map $\cdot: R \otimes M \to M$ (the "action") such that for $r,r' \in R, m \in M$,


*

*$1 \cdot m = m$ (the action is "unital") and 

*$r \cdot (r' \cdot m) = (r\cdot r') \cdot m$ (the action is "associative")
Suppose we copied over this same definition to the case where $R$ is nonassociative. Notice that by two applications of associativity of the action, we get that
$((r_1\cdot r_2) \cdot r_3) \cdot m = (r_1 \cdot (r_2 \cdot r_3))\cdot m$
So the action of $R$ on $M$ acts as though $R$ is associative! More precisely, the action factors through the quotient map $R \to R/A$, where we've modded out by the two-sided ideal $A\subseteq R$ generated by elements of the form $((r_1\cdot r_2)\cdot r_3) - (r_1 \cdot (r_2 \cdot r_3))$, and $R/A$ is associative. So all of the nonassociativity is forgotten when we look at modules.
We could get around this by dropping the requirement that the action be associative. The trouble is that then we're left without any sort of compatibility condition between multiplication in the algebra and the action on the module, so it becomes doubtful that there will be anything interesting to say about these algebraic gadgets.
It is conceivable that there might be more mileage to be gained from the involution or the inner product, but what you're doing is introducing new operations and asserting compatibility relations, so the a priori expectation should be that things will only get more complicated.
Again, here's the link to wikipedia. You mentioned a physics motivation. There is an interesting story to learn about Jordan algebras, a kind of nonassociative ring meant to axiomatize observables in quantum mechanics. And there is a special Jordan algebra of 3 x 3 octonian matrices, but the binary operation of the Jordan algebra does not come directly from the multiplication of octonions.
