What is the formula of this angle? 
Place three congruent equilateral triangles as in figure (the point on side is midpoint of side).
Mark angle is $89.9..$ degree not $90$ degree. It is very strange.

My question. What is the formula of this angle?

 A: 
Putting some labels on the vertices of your triangles, we have congruent equilateral triangles $\triangle ABC,$ $\triangle DEF,$ and $\triangle GHI.$
For simplicity, assume each side of each of these triangles has length $1.$ This does not change the angles but makes it simpler to write the other lengths in the figure.
We then have $AE = \frac12$ and $EP = \frac12\sin 60^\circ < \frac12.$
Since $\triangle DEP$ is a right triangle we have 
$\angle EDP = \arcsin\frac{EP}{DE} = \arcsin EP.$
Since $EP < \frac12$ we also have $\angle EDP < 30^\circ$ and therefore
$\angle HDQ = 60^\circ + \angle EDP < 90^\circ$ and $\sin \angle HDQ < 1.$
We then have  $DH = \frac12$ and $HQ = \frac12\sin \angle HDQ < \frac12.$
Therefore $\angle HGQ = \arcsin\frac{HQ}{GH} = \arcsin HQ < 30^\circ$
and $\angle IGD = 60^\circ + \angle HGQ < 90^\circ.$
In order for one side of an equilateral triangle to be at an exact right angle from the horizontal line that extends side $AB,$ the third vertex of the triangle has to be exactly a distance $\frac12$ from the line $AB.$ But if that vertex is the midpoint of a side of another congruent triangle, and that side does not make exactly a right angle with the line $AB,$ then that vertex will be at a distance less than $\frac12$ from line $AB.$
However, with each triangle that we add to the figure in this way, the third vertex is a little father from line $AB$ than the third vertex of the last triangle,
therefore the angle of the side with line $AB$ is a little closer to a right angle and the distance of its midpoint from line $AB$ is even closer to $\frac12$.
We could keep on adding triangles to the figure like this indefinitely, and the angle between the side and line $AB$ would get closer and closer to a right angle but can never be exactly equal to a right angle. You may find it curious that after the third triangle we are already within one tenth of a degree from a right angle, but we would have gotten that close eventually and we will get much closer if we add a fourth triangle to the figure.
Now suppose we have a figure with $n$ triangles put together this way, and let $h_k$ be the distance from line $AB$ to the third vertex of the $k$th triangle.
That is, $\triangle ABC$ is the first triangle, so $h_1 = 0.$
The $h_k$ terms are a series with $h_2 = EF$ and $h_3 = HQ.$
Let $\theta_k = \arcsin h_k.$
In general, $h_{k+1}$ is half the sine of an angle that is $60$ degrees greater than $\theta_k.$ By the angle addition formula for sines,
$$ h_{k+1} = \frac12\sin\left(60^\circ + \theta_k \right)
=  \frac12\left( \sin 60^\circ\cos\theta_k + \cos 60^\circ\sin\theta_k \right)
=  \frac{\sqrt3}{4}\sqrt{1 - h_k^2} + \frac14 h_k.$$
So $h_2 = \frac{\sqrt3}{4}$ and
$$ h_3 = \frac{\sqrt3}{2}\sqrt{\frac{13}{16}} + \frac14 \frac{\sqrt3}{4}
= \frac{\sqrt{39} + \sqrt3}{16} \approx 0.49857.$$
So
$$ \angle IGD = 60^\circ + \arcsin\left(\frac{\sqrt{39} + \sqrt3}{16} \right)
\approx 89.9051428675277^\circ.$$
Note that we could use the angle addition formula to keep going with a fourth triangle, a fifth triangle, and so forth. The lengths $h_k$ are always algebraic numbers, but of course at the end you have to take an arc sine to get the final angle.
