Infinite Sequence of Inscribed Pentagrams - Where does it converge? If you draw a (not necessarily regular) pentagram, there will be a pentagon-shaped hole in the middle. You can connect points to inscribe a pentagram within that hole, and then inscribe another inside that, etc...
Is anything known about the point to which this process converges?
Plausible but wrong conjectures: center of circumcircle (when it exists), center of gravity of the starting five vertices, point minimizing total distance to the starting five vertices.
Possibly helpful observation: projective transformations preserve lines, so they also preserve the limit point. (Unfortunately, no such transformation can turn a nonsymmetric pentagram into a symmetric one.)
Plausible conjectures are welcomed as comments, but please check some strange/degenerate cases first to make sure they're actually plausible. A solution for any nontrivial class of cases is welcomed as an answer.
 A: Your intuition about a projective transform was correct: a projective transformation relates one iteration to the next, they don't even skip a step. A quick experiment with Cinderella verified this pretty easily. By now I even have a proof, so this answer differs greatly from previous versions.

Mapping corners for one step
This section is a proof that the transformation mapping the outer pentagram to the inner one is in fact a projective transformation. If you prefer to simply take my word for it, feel free to skip to the next section.
A projective transformation is uniquely defined by the images of four points, no three of which may be collinear. So suppose you define the projective map $T$ using the following four points:
\begin{align*}
A_1&\mapsto A_2&
B_1&\mapsto B_2&
C_1&\mapsto C_2&
D_1&\mapsto D_2
\end{align*}
Now you know that $T$ preserves incidences, so (using $\vee$ to denote “join” and $\wedge$ to denote “meet”) you get
\begin{align*}
T(E_2) &= T\bigl((A_1\vee B_1)\wedge(C_1\vee D_1)\bigr)
\\&=\bigl(T(A_1)\vee T(B_1)\bigr)\wedge(T(C_1)\vee T(D_1)\bigr)
\\&=(A_2\vee B_2)\wedge(C_2\vee D_2) = E_3
\end{align*}
You may also know that a projective transformation will preserve cross ratios. Therefore you get $(A_1,B_1;E_2,C_2)=(A_2,B_2;E_3,T(C_2))$. So once you know $A_2$, $B_2$ and $E_3$ (as we do), the point $T(C_2)$ with relations to these has to be fixed. You can construct this point, using the fact that a perspectivity will preserve cross ratios. A perspectivity centered at $D_2$ will map
\begin{align*}
A_1&\mapsto B_2&
E_2&\mapsto C_3&
C_2&\mapsto E_3&
B_1&\mapsto A_2
\end{align*}
This is a reversal of points, but reversing the order of points in the cross ratio will not change its value. So by now you have fixed $E_3$ by intersecting two lines, and now have shown $T(C_2)$ to be the intersection of $A_2\vee B_2$ with $D_2\vee E_2$. Therefore you now know that $T(C_2)=C_3$. With a similar argument you get $T(B_2)=B_3$. And now you can use these to construct $T(E_1)$:
\begin{align*}
T(E_1) &= T\bigl((A_1\vee B_2)\wedge(D_1\vee C_2)\bigr)
\\&=\bigl(T(A_1)\vee T(B_2)\bigr)\wedge(T(D_1)\vee T(C_2)\bigr)
\\&=(A_2\vee B_3)\wedge(D_2\vee C_3) = E_2
\end{align*}
So at this point you know that the fifth point of the outer pentagram will be mapped to the fifth corner of the inner pentagon, even though only the other four points define that map.
Iterative transformations
So what is the transformation which maps the second to the third pentagram? It is the same: as that transformation preserves incidence, it maps the first pentagram and its inscribed pentagram to the second pentagram and its inscribed pentagram, i.e. the third pentagram. So you are in fact asking about the limit of repeated applications of a single transformation.
Limit case
Now consider the limit case. Your projective transformation $T$ is a $3\times 3$ matrix, and its eigenvectors correspond to the fixed points of the transformation. Consider the diagonal form of that matrix. Repeating the transformation means taking powers of that diagonal matrix. But as we are dealing with homogenous coordinates, you can rescale that matrix arbitrarily. This means that in the long run, only the eigenvalue with the maximal absolute value will have any contribution, the other two eigenvalues will converge to zero after rescaling.
So the homogenous coordinates of the convergence point are given by the eigenvector corresponding to the maximal (absolute) eigenvalue.
A: This question was posed a very long time ago, but you may still find this interesting. The construction which you consider was introduced by Richard Schwartz in 1992, and it is called the $\textit{Pentagram Map}$. In his introductory paper, he discusses precisely the phenomenon you observe: pentagons are invariant under this mapping, up to projective equivalence (the proof is a straightforward cross ratio computation). The answer regarding the convergence of this process was provided by Glick in this paper.
The pentagram map has all sorts of curious properties. On hexagons, it is an involution. Even more interestingly, when this map is applied to bigger polygons, it displays quasi-periodic behavior. In fact, it was shown by Ovsienko, Schwartz and Tabachnikov in this paper that the pentagram map is a $\textit{discrete integrable system}$, and it has since been the subject of much research, relating it to the K-dV hierarchy, Poisson structures, cluster algebras, along with many more interesting areas of math, most of which are far beyond me. Anyway, here are a few more papers you may find interesting:


*

*''Integrability of the Pentagram Map'' by Soloviev (link)

*''Pentagram Maps and Refactorization in Poisson-Lie Groups'' by Izosimov and Khesin (link)

*''The Pentagram Map and Y-Patterns'' by Glick (link)


Hope this helps!
