I need help with solving this geometry problem.

Question

Please, help me with this problem. I've been trying to solve it for more than week and I am not even sure it is correct.

Given an obtuse triangle $\triangle ABC$ with obtuse angle $\angle BAC$. The internal bisector of angle $\angle B$ intersects the side $AC$ at point $E$ and the external bisector of angle $\angle A$ at point $O$.

If the center of the circumcircle of $\triangle AEO$ lies on $AB$ and $AE=2$, $EC=3$, find the area of triangle $\triangle ABC$.

Answer 1

Let $K$ (on an extension of $AB$) be the center of the circumcircle of $\triangle OEA$, and let the circumradius be $r$. Let $\alpha = \angle CAO = \angle KAO$ be the external half-angle at $A$. Note that these two parameters are dependent: $\triangle KAE$ is isosceles with legs $r$, base $2$, and base angles $2\alpha$; thus, $r\cos 2\alpha = 1$.

Since $\triangle AKO$ is isosceles with $\angle KOA = \angle KAO = \alpha = \angle CAO$, we have $OK \parallel CA$. This guarantees $\triangle OKB \sim \triangle EAB$, and we see

$$\frac{|OK|}{2} = \frac{|BK|}{|AB|} \implies \frac{r}{2} = \frac{|AB|+r}{|AB|} \implies |AB|= \frac{2 r}{r-2}$$

(Observe that this requires $r > 2$.) By the Angle Bisector Theorem in $\triangle ABC$,

$$\frac{|AE|}{|AB|} = \frac{|CE|}{|CB|} \implies \frac{2}{|AB|}=\frac{3}{|CB|} \implies |CB| = \frac{3r}{r-2}$$

Via the Law of Cosines at (internal) $\angle A$ in $\triangle ABC$,

$$\cos\left(\pi-2\alpha\right) = \frac{|AC|^2+|AB|^2-|BC|^2}{2|AC||AB|} = \frac{r^2-5r+5}{r(r-2)}$$

Recalling that $r\cos 2\alpha=1$ (and, of course, that $\cos(\pi-2\alpha)=-\cos 2\alpha)$), we have

$$-\frac{1}{r} = \frac{r^2-5r+5}{r(r-2)} \implies (r-1)(r-3) = 0 \implies r = 3 \quad \text{(since $r>2$)}$$

Therefore,

$$\cos 2\alpha = \frac{1}{3} \implies \sin 2\alpha = \frac{2\sqrt{2}}{3}$$ $$|AB| = 6$$

so that

$$|\triangle ABC| = \frac{1}{2} |AB||AC| \sin 2\alpha = 10 \sqrt{2}$$