Could a non-linear transformation be orthogonal? Could a non-linear transformation be orthogonal?
Or in other words, could the Jacobian matrix of an orthogonal transformation be dependent of the coordinates (does not contain constant values only)?
 A: As Yashas Samaga commented, this should be a math.se quetsion. But since it's on physiscs, I can get away with a not completely rigorous answer.
If I understand you correctly, you are looking for a (nonlinear) map $f:V\to V$ on a vector space $V$ that preserves the inner product, i.e. 
$$ \left<f(v),f(w)\right>=\left<v,w\right>\,,$$
correct?
If you additionally assume $f$ to be differentiable, then the answer is no: Consider 
$$\left.\frac{\text d}{\text d \epsilon}\left<f(v+\epsilon u),f(w)\right>\right|_{\epsilon=0}$$
for a real number $\epsilon$. By the orthogonality, this is
$$\left.\frac{\text d}{\text d \epsilon}\left<v+\epsilon u,w\right>\right|_{\epsilon=0}=\left<u,w\right>\,,$$
so it is, in particular, independent of $v$. On the other hand, we can evaluate the derivative directly and obtain
$$\left.\frac{\text d}{\text d \epsilon}\left<f(v+\epsilon u),f(w)\right>\right|_{\epsilon=0}=\left<f'(v)\cdot u,w\right>\,,$$
which does depend on $v$, unless $f'(v)=O$ is constant, in which case $f$ is just $f(v)=O \cdot v$ for an orthogonal matrix $O$.
I guess one can extend this argument to avoid the assumption of differentiablity by considering how 
$$\left<f(v+\epsilon u)-f(v),f(w)\right>$$
depends on $v$, but I don't have the time now (and in physics, everything is differentiable anyway...).
A: If the transformation $f : V \to V$ with $V$ real vector space equipped with a symmetric scalar product $\langle\:,\: \rangle$ is surjective the answer is negative: $f$ is necessarily linear.
Let $x,y \in V$ be generic vectors, from
$$\langle f(x),f(y)\rangle = \langle x,y\rangle \tag{1}$$
and the linearity of the left argument in the right hand side, you see that
 $$\langle f(ax+bz),f(y)\rangle =\langle af(x)+bf(z),f(y)\rangle$$
for every reals $a,b$. Namely
$$\langle f(ax+bz) -af(x)-bf(z),f(y)\rangle =0 \:.$$
If $f$ is surjective, we can choose $y\in V$ such that $f(y)=f(ax+bz) -af(x)-bf(z)$, so that 
$$\langle f(ax+bz) -af(x)-bf(z),  f(ax+bz) -af(x)-bf(z)\rangle= 0 \:,$$
i.e.,
$$||f(ax+bz) -af(x)-bf(z)||^2=0$$
As the scalar product is strictly positive, we have that, for every $x,z\in V$ and $a,b \in \mathbb R$,
$$f(ax+bz) -af(x)-bf(z) =0$$
which means that $f$ is linear
$$f(ax+bz) = af(x)+bf(z)\:.$$
As a final comment I stress that, in view of  the polarization identity, the requirement (1) can be weakened into
$$||f(x)||=||x|| \quad \forall x \in V\:,$$
which implies (1). 
This requirement together surjectivity of $f$ implies  linearity of $f$. 
