I actually followed your attempt up to specifying where $P_3$ must lie, but then I solved the problem entirely – with pretty pictures along the way. Saddle up!
Take the unit circle centred on the origin and let the point that may be enclosed be $D$ at $(x,0)$. (I use $x$ because a function in this variable is ultimately going to be found.) The line between the origin $O$ and $D$ naturally splits the circle into upper and lower halves.
This is where symmetry arguments help simplify the problem. If all three points on the circle $A,B,C$ are in the same half, they cannot enclose $D$. Therefore two points are in one half and the third point is in the other half. Without loss of generality, let $A$ and $B$ be in the upper half and $C$ be in the lower half. Define $a=\angle AOD$ and $b=\angle BOD$, i.e. the arguments of $A$ and $B$ when viewed as complex numbers. Furthermore, because $A$ and $B$ can be swapped without affecting the arc where $C$ must be in order for $\triangle ABC$ to enclose $D$, assume $a\ge b$.
With all these boundaries drawn, consider the following picture.
The law of cosines gives $AD=\sqrt{1+x^2-2x\cos a}$. The law of sines then gives $\angle DAO=\sin^{-1}\frac{x\sin a}{\sqrt{1+x^2-2x\cos a}}$, so by angle chasing
$$a'=\angle DOP=\pi-2\angle DAO-a=\pi-2\sin^{-1}\frac{x\sin a}{\sqrt{1+x^2-2x\cos a}}-a$$
The formula for $b'=\angle DOQ$ is similar, only that $a$'s become $b$'s. The range of angles where $C$ can be so that $\triangle ABC$ encloses $D$ is then $\angle POQ=b'-a'$.
$a\in[0,\pi]$ and $b\in[0,a]$, so a double integral gives the "volume" comprised of the enclosing $C$ ranges:
$$\int_0^\pi\int_0^a\left(\left(\pi-2\sin^{-1}\frac{x\sin b}{\sqrt{1+x^2-2x\cos b}}-b\right)-\left(\pi-2\sin^{-1}\frac{x\sin a}{\sqrt{1+x^2-2x\cos a}}-a\right)\right)\,db\,da$$
$$=\int_0^\pi\int_0^a\left(-2\sin^{-1}\frac{x\sin b}{\sqrt{1+x^2-2x\cos b}}+2\sin^{-1}\frac{x\sin a}{\sqrt{1+x^2-2x\cos a}}+a-b\right)\,db\,da$$
This is out of a total volume of $\int_0^\pi\int_0^a\int_0^\pi1\,dc\,db\,da=\frac{\pi^3}2$ (where $c$ is the argument of $C$ when treated as a complex number, ranging from $[0,\pi]$ here), so the probability that $\triangle ABC$ encloses $D$ with the stated range restrictions on the triangle vertices is
$$\frac2{\pi^3}\int_0^\pi\int_0^a\left(-2\sin^{-1}\frac{x\sin b}{\sqrt{1+x^2-2x\cos b}}+2\sin^{-1}\frac{x\sin a}{\sqrt{1+x^2-2x\cos a}}+a-b\right)\,db\,da$$
This probability does not change when we remove the restriction that $a\ge b$. It is multiplied by $\frac34$ when we allow $A,B,C$ to be anywhere along the circle: six of eight half assignments generate the same geometry as the case we considered above and the other two (where all points are in the same half) have probability zero of enclosing $D$. Thus the probability the original question asked for is
$$\frac3{2\pi^3}\int_0^\pi\int_0^a\left(-2\sin^{-1}\frac{x\sin b}{\sqrt{1+x^2-2x\cos b}}+2\sin^{-1}\frac{x\sin a}{\sqrt{1+x^2-2x\cos a}}+a-b\right)\,db\,da$$
We can verify that this gives the correct result when $x=0$. In this case, the two arcsine terms vanish and we are left with
$$\frac3{2\pi^3}\int_0^\pi\int_0^a(a-b)\,db\,da=\frac3{2\pi^3}\cdot\frac{\pi^3}6=\frac14$$
In fact, this allows us to take $a-b$ out of the integrand and highlight that the enclosure probability is less than $\frac14$ when $x>0$:
$$\frac14+\frac3{2\pi^3}\int_0^\pi\int_0^a\left(-2\sin^{-1}\frac{x\sin b}{\sqrt{1+x^2-2x\cos b}}+2\sin^{-1}\frac{x\sin a}{\sqrt{1+x^2-2x\cos a}}\right)\,db\,da$$
A bit more factor handling yields
$$P(x)=\frac14-\frac3{\pi^3}\int_0^\pi\int_0^a\left(\sin^{-1}\frac{x\sin b}{\sqrt{1+x^2-2x\cos b}}-\sin^{-1}\frac{x\sin a}{\sqrt{1+x^2-2x\cos a}}\right)\,db\,da$$
By expanding the Taylor series of the above I sussed out a possible closed form involving the dilogarithm, which was later proved here:
$$\bbox[5px,border:1px solid black]{P(x)=\frac14-\frac3{2\pi^2}\operatorname{Li}_2(x^2)}$$
