A general linear iteative system can be represented as a matrix:

$$(x,y)\mapsto(ax+by,cx+dy)$$

is essentially the same as

$$\left[\begin{array}{cc} a&b\\ c&d\\ \end{array}\right] \left[\begin{array}{c} x\\ y\\ \end{array} \right]$$

which is useful because it can be iterated quickly (matrix exponentiation) and enables various matrix techniques for determining asymptotic behavior and the like. (Of course the number of variables can be increased as needed.)

Is there a similar tool for quadratic iterative systems like

$$(x,y)\mapsto(ax^2+bxy+cy^2,dx^2+exy+fy^2)$$ ? I'm interested in computing the $n$th iterate ($n$ not too small), finding asymptotic behavior, and any other interesting things that can be determined for a given collection of constants $a,b,\ldots$.

My immediate interest (genetics, oddly enough) does not use any of the diagonal terms $x^2,y^2$ so a treatment that ignores them would be fine (though I suspect including is more natural).

-
The complex iteration $z\mapsto az^2$ with $a$ and $z$ both complex would be a special case, but I'm not seeing how to generalize to your form at the moment... – J. M. Sep 29 '11 at 13:29
@J.M.: FWIW the problem at hand has ~5 variables and no diagonal terms, so C isn't 'big enough'. – Charles Sep 29 '11 at 14:10
One indication that it isn't going to be nice and easy is that the Mandelbrot map can be written in that form: $$(x,y,c_1,c_2,u)\to(x^2-y^2+c_1u, 2xy+c_2u, c_1u, c_2u, u^2)$$if we set $u=1$ initially ... – Henning Makholm Sep 29 '11 at 17:46
@HenningMakholm: But the Mandelbrot map needs infinitely many iterations, right? Mine needs only finitely many. (It's not obvious that this is any easier, just different.) – Charles Sep 29 '11 at 23:02
@Charles, the Mandelbrot map is what it is -- just something you can iterate and see what happens. The Mandelbrot set consists of those $(c_1,c_2)$ such that the sequence starting at $(0,0,c_1,c_2,1)$ is bounded -- and boundedness sounds like it naturally belongs among "other interesting things things" you allude to besides asymptotic behavior. One might even say that boundedness is essentially part of "asymptotic behavior". – Henning Makholm Sep 29 '11 at 23:11

Recall that the dynamics of the logistic map $$\lambda\mapsto \lambda x(1-x)$$ can be very complicated ("chaotic"), and can depend sensitively both on the starting value and on the parameter $\lambda$.
In your setting, we can simulate this map by studying $$(x,y)\mapsto (-\lambda x^2 + \lambda xy , y^2),$$ and using a starting value with $y=1$.
However, in the two-variable case, it may be interesting to note that we can project $\mathbb{R}^2$ to projective space (since your polynomial is homogeneous), and the iteration is semiconjugate to a one-dimensional map. More precisely, if we set $p := x/y$, then your map is semiconjugate to the quadratic rational map $$R(p) = \frac{ap^2+bp+c}{dp^2+ep+f}.$$