For a recursion based on $x+y+z+2\sqrt{xy+yz+zx},$ does what happens in $\mathbb Z\left[\sqrt n\right]$ stay in $\mathbb Z\left[\sqrt n\right]$? This problem has a geometric origin which I'll outline below, but I believe the concepts and explanation are algebraic.
Given a function on triples
$$K((x,y,z))=x+y+z+2\sqrt{xy+yz+zx}$$
we build a general recursion as follows.

*

*Start with an integer triple $X=(a,b,c)$, and a set $S=\{a,b,c\}$.


*Compute $k=K(X)$ and add $k$ to $S$.  Writing $X$ as $(X_0,X_1,X_2)$, repeat step 2 for the triples $X=(k,X_1,X_2),(X_0,k,X_2),(X_0,X_1,k)$ respectively.
In general, the members of $S$ will contain nested radicals.  For example, if we start with the triple $(1,2,3)$ and use a restricted recursion  $x_{n+1}=K((1,2,x_n)),x_1=3$ we get
$$
S'=\left\{3,6+2 \sqrt{11},9+2 \sqrt{11}+2 \sqrt{2+3 \left(6+2 \sqrt{11}\right)},\\12+2
   \sqrt{11}+2 \sqrt{2+3 \left(6+2 \sqrt{11}\right)}+2 \sqrt{2+3 \left(9+2
   \sqrt{11}+2 \sqrt{2+3 \left(6+2 \sqrt{11}\right)}\right)},\dots \right\} 
$$
However, Mathematica is able to denest these to
$$
S'=\left\{3,6+2 \sqrt{11},15+4 \sqrt{11},30+6 \sqrt{11},\dots\right\} 
$$
suggesting the conjectures that

*

*starting with initial triple $(1,2,3)$,  $S'\subset S\subset\mathbb Z[\sqrt {11}]$

*starting with integer triple $(a,b,c)$, $S\subset \mathbb Z[\sqrt n]$, where $n=ab+bc+ca.$

Question: is the latter conjecture $S\subset \mathbb Z[\sqrt n]$ true, and if so why?
Bonus question: is there some characterization or invariant of the triples generated in the general recursion?

I've done some hand and computer calculations to understand why the radicals denest in particular cases, but I'm guessing that if the conjecture is true it will be "obvious" to someone more versed in this area.
Some background and observations:

*

*This problem originates in playing with the construction of Apollonian gaskets and the function $K()$ comes from a theorem of Descartes concerning the curvatures of 4 mutually tangent circles.

*In the case that $n=ab+bc+ca$ is a square, then all the numbers generated will be integers. And this leads to integral Apollonian circle packings.  But I have not seen any discussions of the case when curvatures are in some $\mathbb Z[\sqrt n]$.

*see also this demo
 A: We simply define $x_1=3$ and $x_{n+1}=1+2+x_n+2\sqrt{1\cdot2+2x_n+1x_n}$ and thus:
$$x_{n+1}=3+x_n+\sqrt{2+3x_n}$$
We simply prove by induction that $\sqrt{2+3x_n} \in \mathbb{Z}[\sqrt{11}]$. You have already covered the base case. Taking the value $2+3x_n=(a+b\sqrt{11})^2$, we then have:
$$x_{n+1}=3+x_n+a+b\sqrt{11}$$
$$2+3x_{n+1}=11+3x_n+3(a+b\sqrt{11})$$
$$(2+3x_{n+1})=(2+3x_n)+2\cdot3\cdot(a+b\sqrt{11})+3^2$$
$$(2+3x_{n+1})=(a+b\sqrt{11}+3)^2$$
We may also form the general expression for the values in the sequence. Notice that the value $\sqrt{2+3x_{n+1}}=3+\sqrt{2+3x_n}$ and hence, we get:
$$\sqrt{2+3x_{n}}=3(n-1)+\sqrt{2+3x_1}=3(n-1)+\sqrt{11}$$
Hence, we have:
$$x_n=\frac{1}{3} \bigg( (3n-3+\sqrt{11})^2-2\bigg)$$
$$x_n=(3n^2-6n+6)+2(n-1)\sqrt{11}$$
This proves the required with characterization as well.
A: this is "Vieta Jumping" when using the original four variables.
If you have a quadruple $(x,y,z,w) $  in a quadratic number field that satisfies
$$ x^2 + y^2 + z^2 + w^2 - 2(xy+xz+xw +yz+yw+zw) = 0, $$
a revised quadruple can be found merely by jumping one of the entries, as
$$ (2y+2z+2w-x, y, z,w), $$ or
$$ (x, 2x+2z+2w -y, z,w), $$ or
$$ (x, y,2x+2y+2w -z,w), $$ or
$$ (x, y, z, 2x+2y+2z-w). $$
Note that the jumps do not expand the ring at all.
To work out the four jumps from a quadruple $(x,y,z,w),$  it saves space and time to write $s = 2(x+y+z+w)$ so that the four neighbors are $(s-3x,y,z,w)\; \; , \; \; $ $(x,s-3y,z,w)\; \; , \; \; $ $(s,y,s-3z,w)\; \; , \; \; $ $(s,y,z,s-3w)\; \; , \; \; $
You start with $$  (1,2,3, 6 + 2 \sqrt{11}  )$$
gives my $s= 24+4 \sqrt{11}.$
The four neighbors are  $  (21 + 4\sqrt{11},2,3, 6 + 2 \sqrt{11}  ) \; , \; \; \; $ $  (1,18 + 4 \sqrt{11},3, 6 + 2 \sqrt{11}  ) \; , \; \; \;$
$  (1,2,15+4 \sqrt{11}, 6 + 2 \sqrt{11}  ) \; \; , \; \; $ $  (1,2,3, 6 - 2 \sqrt{11}  )$
Let's see, an indefinite quadratic form has an infinite automorphism group, unless very restricted (binary, square discriminant). It is just when the coefficient of every $x_j^2$ is exactly $1$ that we get the very simple jumping behavior, which allows changing one coordinate at a time.
A: Please read my answer to MSE 2953315
"A Diophantine Equation Related to the Markoff Numbers" for some context for the following.
Define a symmetric homogeneous quadratic polynomial
$$ P(x,y,z,w) := x^2+y^2+z^2+w^2-2(xy+xz+yz+wx+wy+wz). \tag{1}$$
Given $\,x,y,z\,$ there are two roots of $\,P(x,y,z,w) = 0.\,$ They are
$$ w_\pm = x+y+z\pm 2\sqrt{xy+yz+zx}. \tag{2}$$
As usual, these two roots satisfy
$$ w_+w_-=x^2+y^2+z^2-2(xy+xz+yz) \tag{3}$$
and
$$ w_++w_-=2(x+y+z). \tag{4}$$
Given any commutative ring $\,R,\,$ define the set
$$ T := \{(x,y,z,w)\in R^4: P(x,y,z,w)=0\}. \tag{5}$$
If $\,(x,y,z,w)\in T\,$ then $\,(x,y,z,2(x+y+z)-w)\in T\,$
because of equation $(4)$. Now define
$$ S := \{\{x,y,z\}\; \text{ where }\; (x,y,z,w)\in T\}. \tag{6}$$
If we suppose $\,\{x,y,z\}\in S\,$ then $\,(x,y,z,w_\pm)\in T\,$
where $\,w_\pm\,$ is from equation $(2)$ and also
$$ (w_\pm,y,z,x),\,(x,w_\pm,z,y),\,(x,y,w_\pm,z)\in T \tag{7}$$
as in step 2 of the recursion since $\,P\,$ is a symmetric function.
In the question we have the special case of
$\,R=\mathbb Z[\sqrt {11}]\,$ and
$\,(1,2,3,6+2\sqrt{11})\in T.\,$
A: Although the other answers here get to the crux of the problem, as the accepted answer I'd like to cite Sarnak, Integral Apollonian Packings, pg 297.
Let the initial triple be $(a,b,c)$, and $\Delta=ab+bc+ca$.
Then all members of $S$ generated by the recursion can be expressed as sums of $a,b,c,$ and $\sqrt{\Delta}$ with integer coefficients.
This is easily shown using some of the observations in the other answers, or the material in the paper.
The nice thing is that the integer coefficients are independent of the initial values. And the generated values are all in $\mathbb Z[\sqrt{\Delta}]$.  This potentially speeds up the generation of Apollonian Gaskets that motivated this question.
