# Convexifying a Concave Polygon by Reflections

Let $$P$$ be a given concave polygon and consider the following procedure. Take the convex hull $$Q$$ of $$P$$ and consider a side $$AB$$ of $$Q$$ which is not a side of $$P$$. Then, let $$P'$$ be the polygon obtained by $$P$$ by replacing the polygonal path of $$P$$ from $$A$$ to $$B$$ which has no vertex in common with those of $$Q$$ other than $$A$$ and $$B$$ with its image in the reflection with respect to $$AB$$. If $$P'$$ is a convex polygon, then the procedure stops, otherwise replace $$P$$ by $$P'$$ and repeat the process.

Does this procedure end after finitely many steps, for every initial concave polygon $$P$$ and every (or at least some) possible choice of the sides with respect to which to take the reflections?

My intuition, shaped by the many concrete examples which I have carefully examined, strongly suggests that the answer is positive, but when I haved tried to sketch a proof, I got completely stuck.

$$\mathbf{Remark}$$. Just note that the polygons $$P$$ and $$P'$$ have the same perimeter, while $$P'$$ has a larger area than $$P$$. Actually, this procedure was described by David in his answer to the post Proving that the regular n-gon maximizes area for fixed perimeter, in which the isoperimetric problem for n-gons is discussed. Even though the eventual finiteness of the procedure is irrelevant to the solution there given, I think this issue has its own interest.

• I think you've given the answer. Each time you do one of these moves, you increase area while maintaining the perimeter. The area can only get so large by the isoperimetric inequality, so there must be a point at which you can no longer increase it. So you can't do one of these reflection moves, so the polygon is not concave. Nov 11, 2021 at 17:13
• @EthanDlugie Dear Ethan, thank you very much for your interest and for your kind comment. Anyway, the issue is not so simple. In principle, the area could keep increasing of smaller and smaller amounts, so converging to a finite limit. We know by intuition that this is not the case, but the real challenge is to prove it! Nov 12, 2021 at 14:04
• Ah, right you are! Nov 13, 2021 at 6:33

You'd think this would be relatively easy to analyze and prove. The flips preserve edge length and order, and increase area. But attempts to proceed further end up in the weeds.

In case of emergency, break glass and go to the internet and find the following paper

Demaine, Gassend, O’Rourke, Toussaint, All Polygons Flip Finitely. . . Right?

Abstract. Every simple planar polygon can undergo only a finite number of pocket flips before becoming convex. Since Erdős posed this finiteness as an open problem in 1935, several independent purported proofs have been published. However, we uncover a plethora of errors, gaps, and omissions in these arguments, leaving only two proofs without flaws and no proof that is fully detailed. Fortunately, the result remains true, and we provide a new, simple (and correct) proof. In addition, our proof handles non-simple polygons with no vertices of turn angle 180◦, establishing a new result and opening several new directions.

The paper goes into extensive detail about the history of the problem and proofs, and before ascending the climb to its own proof finds the dead bodies of flawed proofs of yore on the way.

You'd think that there would be relatively plausible conjectures about the maximum number of reflections required to get to convexity (e.g. $$2n$$ flips for an $$n$$-gon), but there is no such limit, even for quadrangles (https://www.cs.mcgill.ca/~cs507/projects/1998/mas/main2.html).

• Really fascinating, discussion -- thanks!
– Zim
Nov 21, 2021 at 18:03
• @brainjam Thank you very very ... much for your invaluable help and precise references. It is quite moving to know that my conjecture was actually first made by a legend of mathematics as Paul Erdős in 1935 and then (although incorrectly) proved by the other great mathematician Béla Szőkefalvi-Nagy, so that this result is now known as Erdős–Nagy Theorem. Nov 22, 2021 at 13:54
• @brainjam The paper by Demaine, Gassend, O'Rourke and Toussaint is really great! It reconstructs the amazing story of this remarkable result in discrete geometry, by describing all the proofs that were given by many different authors, almost all of which contain some flaw or gap! Every budding mathematician should read this article to realize how easy actually it is to go wrong in mathematics when one relies on allegedly "obvious" facts!!! Nov 22, 2021 at 14:03
• Among the correct and complete proofs the authors survey, I was stunned by the one by Bing and Kazarinoff (which is actually the same proof the authors found by themselves, simply written in a terser style): it is so elegant and brilliant! What a wonder! I am pretty sure that the great Erdős would have said that "it comes from the Book"! Nov 22, 2021 at 14:05
• @Maurizio Barbato - to your second comment, I will never forget when in my first year calculus class the professor said “there are two kinds of obvious statements: those that are true and those that are false.” Nov 22, 2021 at 14:48