An open set invariant under a linear map implies it is an isometry or of finite order? Let $U \subseteq \mathbb R^2$ be an open, bounded, connected subset. Let $A \in \text{SL}_2$ ($A$ is an invertible $2 \times 2$ matrix with determinant $1$) and suppose that $AU = U$.

Must $A$ be either orthogonal or of finite order?

If we assume that $0 \in U$, then I can prove that $A$ is diagonalizable (over $\mathbb C$), with all eigenvalues of modulus $1$. I am not sure if this helps though.
Indeed, we can assume that $B_r(0) \subseteq U \subseteq B_{R}(0)$. Thus for any $x \in B_r(0)$ and for any $k \in \mathbb{Z}$, since $A^k U \subseteq U$, $|A^k x| \le R$. This implies that all orbits of $A$ are bounded, i.e.  $\sup_{k\in\mathbb{Z}}\|A^k x\|<+\infty$ for any $x\in \mathbb{R}^n$, which implies the required assertion about diagonalizablilty.
 A: You don't need to assume that $0\in U$.
Since $AU=U$, all orbits of vectors in $U$ completely lie in $U$. Since $U$ is bounded, no vector in $U$ can have an unbounded orbit (because an unbounded orbit would not lie in $U$). Since $U$ is open, it spans all of $\mathbb R^2$, that is, any vector in $\mathbb R^2$ can be written as linear combination of vectors $u_i$ in $U$. But then for a general vector $v\in\mathbb R^2$, we have
$$\|A^kv\| = \|A^k(\sum_i \alpha_i u_i)\|
= \|\sum_i \alpha_i (A^k u_i)\|
\le \sum_i \|\alpha_i (A^k u_i)\|
= \sum_i |\alpha_i| \|A^k u_i\|$$
and therefore the orbit of $v$ is bounded because all orbits of $u_i$ are.
A: This is false.  That is, there is an element $A\in Sl_2$ which preserves a connected bounded open set $U$, but for which $A$ is neither orthogonal, nor finite order.  In fact, there are many such $A$.
Let's construct some. To begin with, let $B =\begin{bmatrix} \cos \theta & -\sin \theta \\ \sin\theta & \cos\theta\end{bmatrix}$ be a rotation matrix, where $\theta$ is in irrational multiple of $\pi$.  In particular, $B$ has infinite order.  In fact, $\langle B\rangle$, the subgroup of $SO(2)$ generated by $B$, is dense.
Now, $SO(2)$ is not normal in $Sl_2$.  This implies that there is a matrix $C\in Sl_2$ for which $CBC^{-1}\notin SO(2)$.  (For, if $CBC^{-1}\in SO(2)$ for every $C\in Sl_2$, then $C\langle B\rangle C^{-1}\subseteq SO(2)$.  Then, since $SO(2)$ is closed $\overline{ C\langle B\rangle C^{-1}}\subseteq SO(2)$.  But $\overline{C \langle B\rangle C^{-1}} = C\overline{\langle B\rangle}C^{-1} = C SO(2) C^{-1}$.  This shows that $SO(2)$ is normal in $Sl_2$, giving a contradiction.)
Choosing a $C$ with $CBC^{-1}\notin SO(2)$, I claim that $A = CBC^{-1}$ meets all your guidelines.  It's not orthogonal by construction, and since $CBC^{-1}$ and $B$ have the same order, it's not of finite order.
So, we need only construct $U$.  To that end, note that $B$ preserves the ball of radius $1$ centered at $(0,0)$, because $B$ is orthogonal.  Calling this ball $V$, this just means $BV = V$.  Let $U = CV$.
Note that $$AU  = CBC^{-1} CV = CBV = CV = U,$$ so $U$ is preserved.  Since left multiplication by $C$ is an isomorphism, it is, in particular a homeomorphism, so $U$ is connected and open.  Lastly, $U$ is bounded because $C$ has bounded operator norm.
Edit
To answer the question in the comments...
Proposition:  Suppose $A\in Sl_2$ preserves a bounded open set $U$.  Then $A$ is conjugate to an element in $SO(2)$.
Proof:  From Celtschk's answer together with the OPs observation, the hypothesis that $A$ preserves an open set $U$ implies that $A$ is diagonalizable.  Let's assume first that the eigenvalues $r$ and $\frac{1}{r}$ have different magnitudes.  This, in particular, implies they are real numbers.  By swapping $r$ and $\frac{1}{r}$, we may thus assume that $|r| > 1$.  Let $v_1$ be an eigenvector for $r$ and $v_2$ be an eigenvector for $\frac{1}{r}$.  As mentioned in Celtschk's answer, there is a vector $v = a_1 v_1 + a_2 v_2$ with $a_1\neq 0$.  Then, by induction,  $A^n v = r^n a_1 v_1 + \frac{1}{r}^n a_2 v_2 \in U$ for all $n = 1,2,3,...$.  Note that $|A^n v| \geq |r|^n |a_1| \rightarrow \infty$ as $n\rightarrow \infty$.  So, we have contradicted the fact that $U$ is bounded.  This concludes the case where the eigenvalues of $A$ are real.
So, we may assume that the eigenvalues $|r|$ and $|\frac{1}{r}|$ have the same magnitude, which must therefore be $1$.  We will show that the closure of the subgroup generated by $A$, $\overline{\langle A\rangle}$ is compact.  Once we've done this, the Cartan-Iwasa-Malcev theorem tells us that $\overline{\langle A\rangle}$ is conjugate to a subgroup of $SO(2)$, so $A$ is conjugate to something in $SO(2)$.
Now, let's show that $\overline{\langle A\rangle}$ is compact.  Once and for all, choose a (possibly complex) matrix $C$ which diagonalizes $A$, so $CAC^{-1}$ is diagonal.
For a (possibly complex) matrix $D = (d)_{ij}$, let $ \|D\| = max(|d_{ij}|)$.  Note that the inequality $\|AB\| \leq 2\|A\| \|B\|$ holds since any entry of $AB$ is of the form $a_1 b_1 + a_2b_2$ where the $a_i$ are entries in $A$ and the $b_j$ are entries of $B$.
Note also that $\|(CAC^{-1})^n\| = 1$ since $(CAC^{-1})^n$ is diagonal with diagonal entries $\lambda^n$ and $(1/\lambda)^n$.
Thus, we see that $\|A^n\| = \| C^{-1}(CAC^{-1})^n C\| \leq 4\|C^{-1}\| \| (CAC^{-1})^n\| \|C\| = 4\|C^{-1}\| \|C\|$.  That is, every entry of $A^n$ is bounded, independent of $n$.
Thus, the cyclic subgroup generated by $A$ is bounded, and so, therefore is its closure.  That is $\overline{\langle A\rangle}$ is a closed bounded subset of $Sl_2$.  Since $Sl_2$ is closed in $M_2(\mathbb{R})$ (because it's the inverse image of a point under the continuous $\det$ map), $\overline{\langle A\rangle}$ is closed and bounded in $M_2(\mathbb{R})\cong \mathbb{R}^4$.  Thus, it is compact.  $\square$
A: I'm probably (still) missing something, but...
We know $A$ is a real 2x2 matrix of determinant 1, and all eigenvalues have modulus 1, from @celtschk's answer. From this, we know that the eigenvalues are either $\exp(it), \exp(-it)$ or $r, 1/r$, where $t$ and $r$ are real. In the first case, $A$ is a rotation by angle $t$, hence orthogonal. In the second case, the fact that the eigenvalues have modulus $1$ means that they are both $+1$ or both $-1$, so $A$ is either the identity or $-I$.
Probably @celtschk figured all this was trivial and didn't both writing it out, but I've done so for clueless folks like me who come upon this question later.
Post-comment addition
Let $z = \exp(it) = c + i s$, where $c = \cos t, s = \sin t$, so the other eigenvalue is $\bar{z}$. Let $v_1$ be a (complex) eigenvector
for $z$, so
$$
Av_1 = zv_1.
$$
Conjugating both sides gives us
$$
A \bar{v}_1 = \bar{z} \bar{v}_1
$$
(because $A$ is real), so we have an eigenvector $v_2 = \bar{v}_1$ for the other eigenvalue, too.
I believe that if you let
$$
w_1 = \Re(v_1);
w_2 = \Im(v_1)
$$
then you find that $w_1$ and $w_2$ are orthogonal vectors in $\Bbb R^2$, and that
$$
Aw_1 = \cos(t) w_1 + \sin(t) w_2 \\
Aw_2 = \sin(t) w_1 - \cos(t) w_2
$$
so that in the $w$ basis, $A$ is a rotation by $t$...but then it's a rotation by $t$ in any basis.
As I said, I believe this, but I can't actually get the algebra right, and I've got a deadline in a couple of hours, so I have to stop; I hope that this gets you going in the right direction (or lets you see why I'm completely wrong!).
