what is the probability that the circumcircle of 3 point 
Mary  picks any three non-collinear points inside a given circle, what is the probability that the circumcircle of these 3 points will be covered by the original circle?

This is from a  test question a few months ago.I got the result is $\frac{1}{\pi}$,I don't know my result right.What approaches do you think I could take to solving the step?
 A: The probablity $P$ we seek is $\frac{2}{5}$.
WOLOG, choose the coordinate system so that our circle $\mathcal{C}$ is the unit circle centered at origin.
Let 


*

*$\mathcal{S}$ be a big square of side $L$ centered at origin. 

*$A$, $B$, $C$ be three random points selected uniformly from the interior of $\mathcal{S}$. 

*$\mathcal{E}(A,B,C)$ be the condition "the circumcircle for triangle ABC falls inside $\mathcal{C}$".


It is clear the probability $P$ we seek is equal to the conditional probability
$$P 
=
{\bf Pr}\left[\;\mathcal{E}(A,B,C) \mid A,B,C \in \mathcal{C}\;\right]
=
\frac{{\bf Pr}\left[\;
\mathcal{E}(A,B,C)\; \land A,B,C \in \mathcal{C}
\;\right]}{{\bf Pr}\left[\; A, B, C \in \mathcal{C}\; \right]}
$$
Notice if $\mathcal{E}(A,B,C)$ is true, then $A, B, C \in \mathcal{C}$. This leads to
$$P 
= \frac{{\bf Pr}\left[\;\mathcal{E}(A,B,C)\;\right]}{{\bf Pr}\left[\; A, B, C \in \mathcal{C}\; \right]}
= \frac{L^6}{\pi^3}{\bf Pr}\left[\;\mathcal{E}(A,B,C)\;\right]
$$
To compute the probability on RHS of last expression, consider following 
parametrization for $A, B, C$.
$$[ 0,\infty )^2 \times [0,2\pi)^4 \ni ( \rho, r, \theta, \alpha, \beta, \gamma )
\quad\mapsto\quad
\begin{cases}
A &= \rho (\cos\theta,\sin\theta) + r( \cos\alpha, \sin\alpha )\\
B &= \rho (\cos\theta,\sin\theta) + r( \cos\beta,  \sin\beta )\\
C &= \rho (\cos\theta,\sin\theta) + r( \cos\gamma, \sin\gamma)
\end{cases}$$
Geometrically, $\rho(\cos\theta,\sin\theta)$ is the circumcenter and $r$ is the circumradius for the triangle $ABC$. With a little bit of algebra, one can show that
$$dAdBdC = |\sin(\alpha-\beta)+\sin(\beta-\gamma)+\sin(\gamma-\alpha)| \rho r^3 d\rho dr d\phi d\alpha d\beta d\gamma$$
In terms of this parametrization, the condition $\mathcal{E}(A,B,C)$ simply becomes $\rho + r \le 1$.
Let $u = \alpha - \beta, v = \beta - \gamma$ and 
$$\begin{align}
\Phi(u,v) 
&= \sin(u) + \sin(v) - \sin(u+v)\\
&= \sin(\alpha-\beta)+\sin(\beta-\gamma)+\sin(\gamma-\alpha)
\end{align}$$
Integrate over $\phi, \gamma$ first and then $\rho, r$, we obtain:
$$
\begin{align}
P 
&= \frac{L^6}{\pi^3} \int_{\mathcal{E}(A,B,C)} \frac{dA dB dC}{L^6}\\
&= \frac{(2\pi)^2}{\pi^3} 
\left[\int_0^1 \rho \left( \int_0^{1-\rho} r^3 dr \right) d\rho \right]
\left[\int_0^{2\pi}\int_0^{2\pi} |\Phi(u,v)| du dv\right]\\
&= \frac{1}{30\pi}\int_{-\pi}^{\pi}\int_{-\pi}^{\pi} |\Phi(u,v)| du dv
\end{align}
$$
Notice 


*

*$\Phi(-u,-v) = -\Phi(u,v)$,

*$\Phi(u,v) = \Phi(v,u)$

*$\Phi(u,v)$ has same sign as $v$ for $0 \le |v| \le u \le \pi$.


We can split the region of integration for last integral into 4 equal pieces
and get:
$$\begin{align}
P 
&= \frac{2}{15\pi}\int_0^{\pi} \int_{-u}^u |\Phi(u,v)| dv du\\
&= \frac{2}{15\pi}\int_0^{\pi} \int_0^u (\Phi(u,v) - \Phi(u,-v)) dv du\\
&= \frac{2}{15\pi}\int_0^{\pi} \int_0^u 2\sin v ( 1 - \cos u) dv du\\
&= \frac{4}{15\pi}\int_0^{\pi} (1-\cos u)^2 du\\
&= \frac{2}{5}
\end{align}
$$
A: It turns out this question has been asked before on the web and Giaolino
has given an yahoo answer 4 years ago. The probability he get is also $\frac25$. 
Following is my reformatting of his answer for easy reference. All reformatting errors will be mine. 
I have make this answer an community wiki. If anyone notice any mistake in this reformatting, please feel free to fix it.

Let $E$ is be set of triples $A,B,C$ such that the circumcircle lies in the unit disc $D = D(O,1)$.
We want to show that $\int_E dA dB dC = \frac25 \pi^3$.
Let $R$ the radius of the circumcircle of $ABC$ and $G$ its center.
We fix $A$ and we change variables. We suppose that $B,C$ have polar coordinates $(r_b, t_b)$, $(r_c, t_c)$ with $A$ at the origin.
  So $r_b = |AB|$ and $r_c = |AC|$, $t_b = \angle(Ax, AB)$, $t_c = \angle(Ax,AC)$.
We are going to change variables and use $G$, $s_b = \angle(Gx,GB)$, $s_c = \angle(Gx,GC)$ as the new variables, $G$ having polar coordinates $(r_g,t_g)$ still with $A$ at the origin.
Let's compute the Jacobian going from $(r_b,t_b,r_c,t_c)$ to $(R,t_g,s_b,s_c)$.
We have the following formulas, in which we don't need to worry about signs since we'll only use the absolute value of the determinant. They follow from the half-angle property in the circle.
$$\begin{align}
r_b 
&= 2 R \sin\left(\frac{t_g + \pi - s_b}{2}\right) 
 = 2 R \cos\left(\frac{t_g - s_b}{2}\right)\\
r_c 
&= 2 R \cos\left(\frac{t_g - s_c}{2}\right)\\
t_b 
&= \frac{t_g + \pi + s_b}{2} + \frac{\pi}{2} 
 = \frac{t_g + s_b}{2} + \pi\\
t_c 
&= \frac{t_g + s_c}{2} + \pi.\\
\end{align}$$
The partial derivatives are
$$\begin{bmatrix}
 2 \cos\left(\frac{t_g - s_b}{2}\right) & 
-R \sin\left(\frac{t_g - s_b}{2}\right) & 
 R \sin\left(\frac{t_g - s_b}{2}\right) &
 0
\\
 2 \cos\left(\frac{t_g - s_c}{2}\right) &
-R \sin\left(\frac{t_g - s_c}{2}\right) &
 0 & 
 R \sin\left(\frac{t_g - s_c}{2}\right)
\\
0 & \frac12 & \frac12 & 0\\
0 & \frac12 & 0   & \frac12\\
\end{bmatrix}$$
The Jacobian is $R \sin\left(\frac{s_b - s_c}{2}\right) = \frac12 |BC|$.
So $$\begin{align}
dB dC 
&= |AB| |AC| dr_b\,dt_b\,dr_c\,dt_c
= \frac12 R |AB| |AC| |BC| dR\,dt_g\,ds_b\,ds_c\\
&= \frac12 |AB| |AC| |BC| dG\,ds_b\,ds_c
\end{align}
$$ 
  where in the final line we switch back to cartesian coordinates for $G$.
We now exchange the order of integration, fix G and take A in polar coordinates with G as the origin.
Then $\frac12 |AB| |AC| |BC| dA,dG,ds_b,ds_c$
  becomes $\frac12 |AB| |AC| |BC| dR\,dG\,ds_a\,ds_b\,ds_c$.
We turn to polar coordinates in $G$. Set $r = |OG|$ and $g = \angle(Ox, OG)$
We simplify $s_a,s_b,s_c$ in $a,b,c$ and we are reduced to
$$\frac12 \left(8 R^3 \left|
\sin\left(\frac{a-b}{2}\right) 
\sin\left(\frac{b-c}{2}\right)
\sin\left(\frac{c-a}{2}\right)
\right| \right) r\,dR\,dr\,dg\,da\,db\,dc$$
with the conditions $r>0$, $R>0$ and $r+R < 1$, and $a,b,c,g  \in [0,2\pi]$.
$(R,r)$ and $(a,b,c,g)$ can be separated.
The integral in $(R,r)$ gives $1/120$.
We set $u = a - b$, and $v = b - c$. The angular integral can be written as
$$\left|\sin\left(\frac{u}{2}\right) \sin\left(\frac{v}{2}\right) \sin\left(\frac{u+v}{2}\right)\right| dg\,da\,du\,dv$$
The variables $a$ and $g$ can be pulled out giving a factor $4 \pi^2$.
So far we have $\frac{8}{240}(4 \pi^2) = \frac{2}{15}\pi^2$.
So we are left with showing that
$$\int_0^{2\pi}\int_0^{2\pi} 
\left|\sin\left(\frac{u}{2}\right) \sin\left(\frac{v}{2}\right) \sin\left(\frac{u+v}{2}\right)\right| du dv = 3 \pi$$
  which is straightforward.

