Probability of an obtuse triangle in a circle. 
Suppose we randomly pick 2 points A, B within a circle centered
at point O. What is the probability that the triangle formed by ABO is
an obtuse one? (Note that A and B are not exclusively on the
circumference).

And what is the conclusion extended to A, B within a ball instead?
Thanks!
PS this is from a Quant interview.
The following is what I have derived during the exam: (Edited, thank you for your corrections!)
consider the joint probability of x, y coordinate for any point in a unit circle, then $f_{XY}(x, y) = \frac{1}{\pi}$, uniformly distributed inside the circular region. The distance between the point and the center, has thus a distribution $f_Z(z) = 2z$ for $z$ in [0, 1] ($z^2 = x^2 + y^2$).
Randomly pick an A, rotate the circle so that A is right on top of the center O. Suppose now A has a distance $z = a$ $(a > 0)$ away from the origin, then B could only be chosen in the region of

*

*$y > a$

*$y < 0$

*within an inner circle whose diameter is $OA$
Therefore, given that the distance is $z = a$, the probability of ABO being an obtuse triangle is given corresponds to area $\arccos(a) - a\sqrt{1-a^2} +  \frac{a^2\pi}{4} + \frac{\pi}{2}$ (Upper, inner, and lower). By this conditional probability, we could derive the total probability, which is
$$
\int_0^1 \frac{1}{\pi}\left(\arccos(a) - a\sqrt{1-a^2} +  \frac{a^2\pi}{4} + \frac{\pi}{2}\right) \cdot 2a \; da = \frac{3}{4}
$$
But is there an easier way to solve this? This looks like a math competition style of question and I expect some tricks to be at play. Thanks!
 A: The probability that point $A$ is at a distance from the center in $[a,a+da]$ is $2\pi a\,da/\pi=2a\,da$.
The probability that, fixed $A$ as above, $\angle ABO>90°$, is the same as the probability that $B$ lies inside a circle of diameter $OA$, that is $\pi(a/2)^2/\pi=a^2/4$.
Hence the overall probability that $\angle ABO>90°$ is:
$$
p(\angle ABO>90°)=\int_0^1 {a^2\over4}\cdot 2a\,da={1\over8}.
$$
The probability that triangle $ABO$ is obtuse is then:
$$
p(\angle ABO>90°)+p(\angle BAO>90°)+p(\angle AOB>90°)
={1\over8}+{1\over8}+{1\over2}={3\over4}.
$$
EDIT.
For a 3D sphere one can repeat the same argument, obtaining:
$$
p(\angle ABO>90°)=\int_0^1 {a^3\over8}\cdot 3a^2\,da={1\over16}.
$$
A: Did you forget about the possibility of $\angle ABO > 90^\circ$? If you know some geometry, this happens when point B is located inside the circle with diameter $AO$, which easily has the area of $\frac{\pi a^2}{4}$. Now, we can easily calculate the value of $P(\angle ABO > 90^\circ)$. Now by symmetry, the value is the same for $P(\angle BAO > 90^\circ)$. The value of $P(\angle ABO > 90^\circ)$ is trivially $0.5$ as you said, and all those probabilities are exclusive and add up to the answer you need.
This answer extends naturally to the 3D case.
A: I first compute the probability $p$ that the triangle $\triangle:=OAB$ is acute.
When$|OA|=r$ we may assume $A=(r,0)$. The point $B=(u,v)$ then has to satisfy $0<u<r$, and $B$ must not lie in the small disc with diameter $OA$. The feasible area for $B$ therefore is
$$b(r):=2\int_0^r\sqrt{1-x^2}\>dx-{\pi r^2\over4}\ .$$
Since $|OA|$ is distributed with density $2r$ with respect to $dr$ we therefore obtain
$$p=\int_0^1 {b(r)\over\pi}\>2r\>dr=\ldots={1\over4}\ .$$
It follows that $\triangle$ is obtuse with probability ${3\over4}$.
A: Using $$f(a)=\left(2\pi \arccos(a) - a\sqrt{1-a^2} + \frac{1}{2}\right) \, \frac{4}{\pi}\sqrt{1 - a^2}$$
$$I=\int_0^1 f(a)\, da=\frac{5}{2}-\frac{1}{\pi }+\frac{\pi ^2}{2}$$ is not too difficult provided that we have some time.
For an approximation, write for example
$$I_n=\int_0^{\frac 12} f(a)\, da+\int^1_{\frac 12} f(a)\, da$$ and use Taylor series of $f(a)$ around $a=0$ to $O(a^{n+1})$ and $a=1$ to $O((1-a)^{n+1})$.
$$I_0=\frac{1}{\pi }+2 \pi\qquad \qquad\text{relative error = 7.23 %}$$
$$I_1=1-\frac{7}{6 \pi }+2 \pi\qquad \qquad\text{relative error = 2.88 %}$$
A: Wherever A and B are, one of them is going to be at least as far away from O as the other. Which point has this property is even odds and independent of the obtusity of the triangle, so assume WLOG that A is at least as far away as B.
Once you've "rotated" the point A to be directly above O, you know B is somewhere in (or on) the concentric circle passing through A, with a uniform distribution.
You know where A is relative to this configuration, but you aren't sure about where B is. Regardless, scale the configuration so that this new A-circle has a radius of $\frac{1}{\sqrt{\pi}}$ (and thereby an area of $1$). This will preserve the obtusity of ABO and avoids the need to integrate wrt the radius $a$.
Since the interior angle sum of ABO is $180^\circ$, we know there's at most one obtuse angle ($\ge 90^\circ$) in the triangle.
As Bimo points out:

*

*Our first case, $\angle BOA\geq 90^\circ$, corresponds to B in the
lower half of the A-circle.

*Our second case, $\angle ABO\geq
   90^\circ$, means B is in a smaller circle, half as tall as the
A-circle and fitting neatly in its upper half (by Thales' theorem).

*The third case one might consider, $\angle OAB\geq 90^\circ$, is
impossible, since this would imply that B were above A (not directly
above).

*If none of these are true, B must be elsewhere in the circle.

Adding the (almost) disjoint first and second cases gives a probability of $\frac{1}{2}+\frac{1}{4}$ quite simply.
Integration is certainly an appropriate tool here but, as your instincts suggest, should not be used where easier, less mistake-prone solutions can be found.
