How many vertices so that the number of pentagons equals the number of triangles? Question: A set of points is chosen on the circumference of a circle so that the
number of different triangles with all three vertices among the points is equal to the
number of pentagons with all five vertices in the set. How many points are there?
Solution
$$\frac{x!}{3!(x-3)!}=\frac{x!}{5!(x-5)!}\\
\frac{x(x-1)(x-2)(x-3)!}{3!(x-3)!}=\frac{x(x-1)(x-2)(x-3)(x-4)(x-5)!}{5!(x-5)!}\\
\frac{x(x-1)(x-2)}{6}=\frac{x(x-1)(x-2)(x-3)(x-4)}{120}\\
6x(x-1)(x-2)(x-3)(x-4)=120x(x-1)(x-2)$$
Hence, $x=1, x=1, x=2, x=-1, x=8$. Since a pentagon has $5$ vertices, the only plausible solution is 8. I couldn't find the solution online, just making sure.
 A: Yes, this reasoning is correct. (Of course, $(n - 5)!$ is not defined for integers $\leq 4$, so we can discard those solutions outright for that reason, too.)
Alternatively, if there are $3 + 5 = 8$ vertices, then we can define a natural bijection $$\{\textrm{triangles}\} \leftrightarrow \{\textrm{pentagons}\}$$ taking a triangle, throwing out its vertices, and forming a pentagon from the remaining five, so, e.g., mapping the blue triangle to the red pentagon below.

This reflects the symmetry ${8 \choose 3} = {8 \choose 5}$, which is a special case of the general identity ${n \choose k} = {n \choose {n - k}}$.
A: Until the last paragraph, your computations look fine, though the exposition leaves much to be desired (that is, you have done essentially nothing to explain the why of your computations).  Moreover, in your computation, you should connect one line to the next.  As I read from top to bottom, are the equations you give related?  If so, explain how (e.g. note that the equation above implies the equation below).
Finally, as hinted above, I am a bit confused by your last paragraph.  Part of the problem is a typo, which I think can be corrected.  The other problem is that you appear to be claiming that $x$ takes on many values at once.  This is a more severe problem.  Were it me, I would probably write something like the following:


Question: A set of points is chosen on the circumference of a circle so that the number of different triangles with all three vertices among the points is equal to the number of pentagons with all five vertices in the set. How many points are there?

Solution:  Suppose that there are $n$ points on the circle.[1]  A triangle may be formed by choosing $3$ of these $n$ points, hence there are $\binom{n}{3}$ different ways of forming a triangle.  Similarly, there are $\binom{n}{5}$ ways of forming different pentagons.[2]  It is therefore necessary to find $n$ such that
$$ \binom{n}{3} = \binom{n}{5}. $$
Thus
\begin{align}
\binom{n}{3} = \binom{n}{5}
&\implies \frac{n!}{3!(n-3)!} = \frac{n!}{5!(n-5)!} \\
&\implies \frac{n(n-1)(n-2)(n-3)!}{3!(n-3)!} = \frac{n(n-1)(n-2)(n-3)(n-4)(n-5)!}{n!(n-5)!} \\
&\implies \frac{n(n-1)(n-2)}{3!} = \frac{n(n-1)(n-2)(n-3)(n-4)}{5!} \\
&\implies 5! n (n-1)(n-2) = 3! n (n-1)(n-2)(n-3)(n-4). \\
\end{align}
Both the left- and right-hand sides of last equation are zero if $n \in \{0,1,2\}$.[3]  If $n$ is not one of these, then the factors on the left and right may be canceled to obtain
\begin{align}
&5! n (n-1)(n-2) = 3! n (n-1)(n-2)(n-3)(n-4) \\
&\qquad\qquad\implies 5! = 3! (n-3)(n-4) \\
&\qquad\qquad\implies 20 = (n-3)(n-4) \\
&\qquad\qquad\implies n^2 - 7n - 8 = (n+1)(n-8)= 0.
\end{align}
This final equation is solved by $n\in\{-1,8\}$, thus the number of points on the circle must be among the set
$$ n \in \{-1, 0, 1, 2, 8\}. $$
As $5$ of the $n$ must be selected to produce a pentagon, it must be the case that $n \ge 5$.  Among the possible values of $n$, only $n=8$ satisfies this condition.  Therefore there must be $8$ points on the circle.[4]
Notes:
[1] The number of points on the circle is a natural number.  To me, the variable $x$ looks like a real number, so I, personally, would prefer to name the variable something like $n$.  This is entirely a matter of taste.
[2] This first paragraph explains to the reader what is going on.  It provides setting and context.
[3] Again, I like to provide some signposts and let the reader know what I am doing.  It doesn't have to take up a lot of space, but I think that it is helpful.  A wall of computations is hard to read (already, I think that I would likely leave out half of the steps shown).
[4] I find it nice to end an argument with a short, declarative statement.  You have done something.  Say so. ;)

I would also like to point out that there is a rather nice result sneaking around here which you might consider trying to prove:  fix some value of $n$ and suppose that $k_1$ and $k_2$ are distinct natural numbers.  Then
$$ \binom{n}{k_1} = \binom{n}{k_2} $$
if and only if $k_1 + k_2 = n$.  In a sort of geometric way, this corresponds to a symmetry of Pascal's triangle (another thing which is worth investigating in the context of combinations).
