Two vertices of a triangle being fixed, find the locus of the third vertex such that the triangle has its circumcenter inside the incircle Recently I played a little bit around with GeoGebra and I constructed the in- and circumcircle of a $\triangle ABC$ with $A=(0,0)$ and $B=(1,0)$ and I asked myself if it is possible to construct the area where all the points are for which the center of the circumcircle is still inside the incircle. With the trace option I figured that it almost looks like a circle but I don't think it is one so how could one approach a construction of it? I came to no real approach or start.

I baptize this curve with the name ,,Rocklage-curve". It is probably possible to create similar surfaces in other cases.
 A: I will first find a condition the side lengths $a,b,c$ must satisfy for the circumcentre to lie within the incircle. In trilinear coordinates $x:y:z$ the incircle has equation
$$ayz+xbz+xyc-(ax+by+cz)\left(\frac{x(b+c-a)^2}{4bc}+\frac{y(c+a-b)^2}{4ca}+\frac{z(a+b-c)^2}{4ab}\right)=0$$
Now substitute the circumcentre's coordinates $a(b^2+c^2-a^2):b(c^2+a^2-b^2):c(a^2+b^2-c^2)$ into this equation and factor over $\mathbb Q(\sqrt2)$. The factor that matters is
$$a^3+b^3+c^3-(a^2(b+c)+b^2(c+a)+c^2(a+b))+2\sqrt2abc$$
Since the circumcenter lies in the incircle when $a=b=c=1$, our desired condition is when the above expression is nonpositive:
$$a^3+b^3+c^3-(a^2(b+c)+b^2(c+a)+c^2(a+b))+2\sqrt2abc\le0$$
$$\iff\frac1{2abc}(a+b-c)(b+c-a)(c+a-b)\ge\sqrt2-1$$
You want the region for $C=(p,q)$ where the condition holds for $\triangle ABC$. In this triangle $c=1$, $a=\sqrt{p^2+q^2}$ and $b=\sqrt{(1-p)^2+q^2}$. Substituting these into the condition and plotting in the $pq$ plane reveals a very decidedly non-circular shape:


Numerical integration based on the cylindrical algebraic decomposition in Mathematica reveals the area and perimeter of this shape as
$$A=0.9271379517\dots$$
$$L=3.4431978101\dots$$
A: Here is a solution based on ratio $\frac{r}{R}$, where $r$ is the inradius and $R$ the circumradius.
See Geogebra animation here. A static view is available at the bottom of this answer.

*

*Let us recall a relationship called "Euler formula":

$$d^2=R^2-2Rr$$
connecting $r$ and $R$ with $d$ defined as the distance between the incenter $I$ and the circumcenter $O$ of a triangle.
As your inclusion condition is:
$$d \le r,\tag{1}$$
your issue is equivalent to condition :
$$R^2-2Rr \le r^2 \ \iff \ (R-r)^2-r^2 \le r^2$$
Otherwise said, equivalent to :
$$\frac{r}{R} \ge \sqrt{2}-1\tag{2}$$

*

*Besides, there exists another formula involving ratio $\frac{r}{R}$, which is

$$\frac{r}{R}= \cos(\hat{A})+\cos(\hat{B})+\cos(\hat{C})-1 \tag{3}$$
where $\hat{A},\hat{B},\hat{C}$ are resp. the angles in vertices $A,B,C$ ; see here at paragraph "further conditional identities".
Connecting $(2)$ and $(3)$, the inequality implicitly describing the locus of point $C$ is
$$\cos(\hat{A})+\cos(\hat{B})+\cos(\hat{C}) \ge \sqrt{2}\tag{4}$$
It remains to express (4) in terms of the coordinates $(x,y)$ of $C$.
Intermediate expressions are the sidelengths $a=BC$ and $b=AC$ of triangle $ABC$ given by :
$$\begin{cases}a&=&\sqrt{(x-1)^2+y^2},\\  b&=&\sqrt{x^2+y^2}\end{cases}\tag{5}$$
Relationships (5) are useful in the following expressions of the 3 terms in the LHS of (4) :
$$\begin{cases}\cos(\hat{A})&=&\frac{x}{b}\\
\cos(\hat{B})&=&\frac{(1-x)}{a}\\ \cos(\hat{C})&=&\frac{a^2+b^2-1}{2ab}\end{cases}\tag{6}$$
(the last formula comes from cosine rule : $1=a^2+b^2-2ab \cos \hat{C}$)
It remains to "plug" formulas (5) and (6) into (4) to get the implicit equation of the frontier of the locus. Using a CAS like Geogebra we obtain the following shapes which coincides with the locus given in the answers of Parcly Taxel and Robjohn (with $a=f(x,y)$ and $b=g(x,y)$) ; [the locus itself is the two interior parts of these curves].

A: In this answer, it is shown that the distance $d$ between the incenter and circumcenter is
$$
d^2=R(R-2r)\tag1
$$
where $R$ is the circumradius and $r$ is the inradius.
For the circumcenter to be within the incircle, we would need $d\lt r$, that is
$$
R(R-2r)\lt r^2\tag2
$$
Therefore,
$$
\left(\frac Rr\right)^2-2\frac Rr-1\lt0\tag3
$$
which is satisfied if $1-\sqrt2\lt\frac Rr\lt1+\sqrt2$. Since $r,R\gt0$, we need
$$
\frac Rr\lt1+\sqrt2\tag4
$$
From $(3)$ and $(4)$ in the same answer, we get
$$
\begin{align}
\frac Rr
&=\frac{\frac{abc}{4A}}{\frac{2A}{a+b+c}}\tag{5a}\\
&=\frac{abc(a+b+c)}{8A^2}\tag{5b}\\
&=\frac{abc(a+b+c)}{8s(s-a)(s-b)(s-c)}\tag{5c}\\
&=\frac{2abc}{(b+c-a)(a-b+c)(a+b-c)}\tag{5d}
\end{align}
$$
Explanation:
$\text{(5a):}$ $(3)$ and $(4)$ from this answer
$\text{(5b):}$ simplify
$\text{(5c):}$ Heron's Formula (proven in this answer)
$\text{(5d):}$ $2s=a+b+c$
Fix the endpoints of $c$ and plot locus of the remaining vertex of the triangles with sides $a$ and $b$ where
$$
\frac{2abc}{(b+c-a)(a-b+c)(a+b-c)}\lt1+\sqrt2\tag6
$$

