Area of equilateral triangle inscribed in right triangle An Isosceles right triangle  $ABC$  , $AB=BC= 4 cm$
Point $p$ is a midpoint of $BC$ , points $q$, $s$  lies on $AC$,$AB$ respectively  , such that the triangle $pqs$ is  an equilateral triangle ; 
what is the area of triangle $pqs$ 

 A: Rotate $\Delta Bps$ clockwise $60^\circ$ about $p$ as shown, to form $\Delta B'pq$.
Using $\Delta B'Cp$, find that $B'C=2\sqrt3~\mathrm{cm}$.
Using Sine Law on $\Delta B'Cq$, find that $\dfrac{B'q}{\sin 15^\circ}=\dfrac{B'C}{\sin 105^\circ}$, so $B'q=(4\sqrt3-6)~\mathrm{cm}$.
Use Pythagoras' theorem on $\Delta B'pq$ to find that $pq=2\sqrt{22-12\sqrt3}~\mathrm{cm}$.
Therefore the area is $\dfrac12a^2\sin60^\circ=(22\sqrt3-36)~\mathrm{cm}^2$.

A: Based on your description, let's define:
$$ \begin{align}
\mbox{Hypotenuse AC}:\; y &= -x +4 \\
p &= (2,0) \\
q &= (x, y=-x+4) \\
s & = (0,h)
\end{align}$$
The three sides are equal, hence:
$$ \begin{align}
\mbox{sp = pq} \Rightarrow 
4+h^2 &= (x-2)^2 + (x-4)^2 \quad (1)\\
\mbox{sp = sq} \Rightarrow 
4+h^2 &= x^2 + (h+x-4)^2, \quad(2) \\
\mbox{pq = sq} \Rightarrow 
(x-2)^2 + (x-4)^2 &=  x^2 + (h+x-4)^2. \quad(3)\\
\end{align}$$
There are 3 equations with only 2 variables $x$ and $h$ but only two equations are needed to solve for the variables. Let's say we solve $(2)$ and $(3)$ for $x$ and $h$. Then we plug the solution in to $(1)$ and if $(1)$ is not satisfied, then the problem has no solution. 
From $(3)$:
$$ x = \frac{4 + 8h -h^2}{4 + 2h}. \quad(4) $$
Plugging $(4)$ into $(2)$ will result in an equation in $h$ alone. This equation can be solved numerically. Once $h$ is obtained, $x$ will be found by $(4)$.
The next step is to plug the results for $x$ and $h$ into $(1)$. If the equality fails, the problem has no solution.
If the problem has a solution, then the area of the triangle $spq$ is given by
$$ A = \frac{\sqrt3}{4}(sp)^2 =  \frac{\sqrt3}{4}(4+h^2). $$
