Generalizations for non-linear reflections In my geometry class, we're currently learning about isometries and different types of transformations. In particular, we're asked to try to generalize what a reflection transformation does to a point $p$ over a line $\ell$ (Shown in the figure below). This got me wondering how a reflection would look like over a circle, parabola, or any function and what a generalization for such a reflection would be. How would I go about trying to generalize these reflections? Thanks! 

 A: Here is an answer for the reflection with respect to circles.
There is a natural reflection transform with respect to circles provided by the "inversive transform" (or "inversion") which, in its normalized form (with a unit radius circle) can be described in such a way :
$$M(x,y) \mapsto M'(X,Y) \ \ \iff \ \ \vec{OM}.\vec{OM'}=1 \ \ \iff \ \  \ \vec{OM'}=\dfrac{1}{OM^2}\vec{OM}$$
giving formulas :
$$\begin{cases}X&=&\dfrac{x}{x^2+y^2}\\Y&=&\dfrac{y}{x^2+y^2}\end{cases}\tag{1}$$
3 basic properties of inversion :


*

*The closer point $M$ is from the center, the fartest is point $M'$.

*(involutivity) If the image of $M$ is $M'$, the image of $M'$ is $M$.

*The points of the unit circle are invariant. (It is called the "circle of inversion"). 

Fig. 1 : The fish is "more or less mirrored" into the circle of inversion (fish bowl !) into another  deformed fish ; note that its pectoral fin, close to the border is almost perfectly mirrored into the circle, whereas the tail fin has undergone a magnification by $\approx 2$.
For a thorough introduction, see this excellent document. See as well the first image on this excellent question (it is mine ! :)).
Here is the mathematical explanation of the connection with symmetry. It needs some knowledge of calculus and linear algebra.
A first common feature  with symmetry is involutivity. But there is more to come.
Indeed, the jacobian of transformation (1) is :
$$J=\begin{pmatrix}\partial X/\partial x&\partial X/\partial y\\ \partial Y/\partial x&\partial Y/\partial y\end{pmatrix}$$
$$=\dfrac{1}{x^2+y^2}\begin{pmatrix}-\dfrac{x^2-y^2}{x^2+y^2}&\dfrac{2xy}{x^2+y^2}\\\dfrac{2xy}{x^2+y^2}&\dfrac{x^2-y^2}{x^2+y^2}\end{pmatrix} =\dfrac{1}{x^2+y^2}\begin{pmatrix}-\dfrac{1-t^2}{1+t^2}&\dfrac{2t}{1+t^2}\\ \dfrac{2t}{1+t^2}&\dfrac{1-t^2}{1+t^2}\end{pmatrix}$$
(in the second expression $t:=\dfrac{y}{x}$) 
in which we recognize, up to a scaling given by the front fraction, the classical matrix of a symmetry:
$$\begin{pmatrix}-\cos \theta&\sin \theta\\ \ \ \ \sin \theta&\cos \theta\end{pmatrix}$$
using tangent half-angle formulas where $t$ is interpreted as $\tan \dfrac12 \theta$.
Why is the symmetry appearing at this level ? Because the jacobian provides the closest linear approximation to the given transform.
How can we interpret homothetic factor $\dfrac{1}{x^2+y^2}$ ? It deals with the change of scale between the object and its image by the inversive transform. This change of scale being huge if the object gets far away (in this case this factor becomes very small) and vice versa. 
