$M$ is a point in an equalateral $ABC$ of area $S$. $S'$ is the area of the triangle with sides $MA,MB,MC$. Prove that $S'\leq \frac{1}{3}S$. 
$M$ is a point in an equilateral triangle $ABC$ with the area $S$. Prove that $MA, MB, MC$ are the lengths of three sides of a triangle which has area $$S'\leq \frac{1}{3}S$$


We can assume that side of a triangle $ABC$ is $1$. Further let $CM =x$ and $\angle KCM =\gamma$. Rotate $M$ around $C$ for $-60^{\circ}$ in to $F$. Then the area of triangle $AMF$ is the one we are looking for and it's area is area of $AMCF$ minus area of equilateral triangle $CFM$, so $$4S' = -x^2\sqrt{3}+2x\sin (60^{\circ}+\gamma)$$
and this should be easy to calculate that is less than ${\sqrt{3}\over 3}$.
If we see $S'$ quadratic function on $x$ we get:
$$ 4S'\leq {1\over \sqrt{3}}\sin (60^{\circ}+\gamma)\leq {1\over \sqrt{3}}$$
From here we can see that equality is achieved iff $\gamma = 30^{\circ}$ and $x= {\sqrt{3}\over 3} = {2\over 3}v$ where $v$ is altitude of triangle $ABC$. That is, equality is achieved iff $M$ is gravity centre of $ABC$.

I'm interested in different solutions (for example without trigonometry).
 A: Reflect $M$ with respect to the sides of $ABC$. You get an hexagon whose area is $2S$:

The hexagon can be decomposed as the union between $A'B'C'$ (whose side lengths are $\sqrt{3}MA,\sqrt{3}MB,\sqrt{3}MC$) and the isosceles triangles $CA'B',BC'A',AB'C'$. It follows that
$$ 2S = \frac{\sqrt{3}}{4}(AM^2+BM^2+CM^2)+ 3 S'$$
where $S'$ is the area of a triangle with side lengths $MA,MB,MC$. By Weitzenbock's inequality $AM^2+BM^2+CM^2 \geq 4\sqrt{3}S'$, hence $S'\leq \frac{S}{3}$ as wanted.
A: Solution using complex numbers. Copied from a deleted question per request.
( update - I have added another solution using circle inversion at end)

Solution 1 - using complex numbers.
Choose a coordinate system so that triangle $ABC$ is lying on the unit circle centered at origin and $A$ on the $x$-axis. Let $a = AM$, $b = BM$, $c = CM$ and $S'$ be the area of a triangle with sides $a,b,c$. In this coordinate system, $S = \frac{3\sqrt{3}}{4}$, we want to show $S' \le \frac{\sqrt{3}}{4}$. Using Heron's formula, this is equivalent to
$$16S'^2 = (a^2+b^2+c^2)^2 - 2(a^4+b^4+c^4) \stackrel{?}{\le} 3$$
Identify euclidean plane with the complex plane. The vertices $A,B,C$ corresponds to $1, \omega, \omega^2 \in \mathbb{C}$ where $\omega = e^{\frac{2\pi}{3}i}$ is the cubic root of unity. Let $z$ be the complex number corresponds to $M$ and $\rho = |z|$, we have
$$
\begin{cases}
a^2 = |z-1|^2 = \rho^2 + 1  - (z + \bar{z})\\
b^2 = |z-\omega|^2 = \rho^2 + 1 - (z\omega + \bar{z}\omega^2)\\
c^2 = |z-\omega^2|^2 = \rho^2 + 1 - (z\omega^2 + \bar{z}\omega)
\end{cases}
\implies
a^2 + b^2 + c^2 = 3(\rho^2 + 1)
$$
Thanks to the identity $\omega^2 + \omega + 1 = 0$, all cross terms involving $\omega$ explicitly get canceled out.   
Doing the same thing to $a^4 + b^4 + c^4$, we get
$$\begin{align}a^4 + b^4 + c^4 
&= \sum_{k=0}^2 (\rho^2 + 1 + (z\omega^k + \bar{z}\omega^{-k}))^2\\
&= \sum_{k=0}^2\left[ (\rho^2 + 1)^2 + (z\omega^k + \bar{z}\omega^{-k})^2\right]\\
&= 3(\rho^2 + 1)^2 + 6\rho^2\end{align}$$
Combine these, we obtain
$$16S'^2 = 3(\rho^2+1)^2 - 12\rho^2 = 3(1 - \rho^2)^2$$
Since $M$ is inside triangle $ABC$, we have $\rho^2 \le 1$. As a result,
$$S' = \frac{\sqrt{3}}{4}(1-\rho^2) \le \frac{\sqrt{3}}{4} = \frac13 S$$

Solution 2 - using circle inversion.
Let $a = AM, b = BM, c = CM$ again. Let $\Delta(u,v,w)$ be the area of a triangle with sides $u,v,w$. In particular, $S = \Delta(1,1,1)$ and $S' = \Delta(a,b,c)$. We will use the fact  $\Delta(u,v,w)$ is homogeneous in $u,v,w$ with degree $2$.
Consider the circle inversion with respect to a unit circle centered at $A$.
Under such an inversion, $B,C$ get mapped to itself while $M$ mapped to a point $M'$ with $$AM' = \frac{1}{a}, BM' = \frac{b}{a}, CM' = \frac{c}{a}$$
We can decompose the quadrilateral $ABM'C$ in two manners. $\triangle ABC + \triangle BM'C$ and $\triangle ABM' + \triangle AM'C$. This leads to
$$\begin{align}
&\verb/Area/(ABC) + \verb/Area/(BM'C) = \verb/Area/(ABM') + \verb/Area/(AM'C)\\
\iff &
S + \Delta(1,\frac{b}{a},\frac{c}{a}) = \Delta(1,\frac{b}{a},\frac{1}{a}) + \Delta(1,\frac{c}{a},\frac{1}{a})\\
\iff &
Sa^2 + S' = \Delta(1,a,b) + \Delta(1,a,c)
\end{align}
$$
By a similar argument, we have
$$
Sb^2 + S' = \Delta(1,b,a) + \Delta(1,b,c)\quad\text{ and }\quad
Sc^2 + S' = \Delta(1,c,a) + \Delta(1,c,b)
$$
Summing these three equalities together and notice
$$\Delta(1,a,b) + \Delta(1,b,c) + \Delta(1,c,a)
= \verb/Area/(ABM) + \verb/Area/(BCM) + \verb/Area/(CAM)
= S$$
We obtain
$$3S' = 2S - S(a^2+b^2+c^2)$$
For any $\triangle XYZ$ and point $P$ in the plane, we know the expression $XP^2 + YP^2 + ZP^2$ is minimized when $P$ is the centroid of $\triangle XYZ$. For an  equilateral triangle of side $1$, the centroid is at a distance $\frac{1}{\sqrt{3}}$ from the vertices. This implies $a^2 + b^2 + c^2 \ge 1$.
As a result, $3S' \le S$ and we are done.
