A definition of "algebra" (that is, an associative algebra, in the sense of ring theory) generally requires a commutative base ring. But there are cases where it's reasonable to consider algebras over general unital rings.
For example, suppose $R$ is a commutative ring and consider $R^2$. If we multiply ordered pairs coordinatewise, $R^2$ has an obvious $R$-algebra structure: it's an $R$-module in which the abelian group structure is extended to a ring whose multiplication is bilinear, or equivalently (*) $r \cdot \alpha \beta = (r \cdot \alpha) \beta = \alpha (r \cdot \beta)$ (writing $\cdot$ for the $R$-module action).
Now let $R$ be a general unital ring. $R^2$ is a left $R$-module, but the ring multiplication is not generally $R$-bilinear because (*) fails: $r \cdot \alpha \beta \neq \alpha (r \cdot \beta)$. So the definition of an algebra suggested above does not work. On the other hand, $R^2$ does have a ring structure and a module structure (on each side), and those structures seem compatible, so there should be some good way to refer to $R^2$ as an algebra over $R$.
So here's a definition: a "two-sided algebra" $M$ over a ring $R$ is a bimodule $_R M _R$ (including the requirement that $(r \cdot \alpha) \cdot s = r \cdot (\alpha \cdot s)$) with a ring structure on the abelian group such that $r \cdot \alpha \beta = (r \cdot \alpha) \beta$ and $\alpha \beta \cdot r = \alpha (\beta \cdot r)$, for all $\alpha, \beta \in M, r \in R$. (The term "bialgebra" would be nicer but seems well established with a different meaning.)
There's an obvious two-sided algebra structure on $R^2$. If the left and right actions in a two-sided algebra $M$ are the same, then $r \cdot \alpha \beta = \alpha \beta \cdot r = \alpha(\beta \cdot r) = \alpha(r \cdot \beta)$, so (*) is satisfied and the ring multiplication on $M$ is bilinear. So in the case of $R^2$, the two-sided algebra structure over an arbitrary ring generalizes the algebra structure over a commutative ring.
Other good examples are general product rings $R^I$, matrix rings $M_n(R)$, and polynomial rings $R[x]$. The two-sided algebras over $R$ (which we could call $_R \text{Alg}_R$) seem to form a category much like $R$-algebras in the commutative case: it has finite products, an initial object $R$, and a terminal object $0$.
Finally, my questions:
Is the above correct?
Are there any other approaches to this problem?