How to transform regular polygon in irregular polygon? Let $P_1$ be a irregular polygon with $n$ sides, $P_1$ is given by
\begin{equation}
P_1 = \begin{bmatrix}
x_1 \quad x_2 \quad \ldots \quad x_n \\
y_1 \quad y_2 \quad \ldots \quad y_n
\end{bmatrix}.
\end{equation}
Let $C_x$ and $C_y$ be center of $P_1$ in $x$ and $y$ axes, respectively. And let $P^{*}_1$ be a regular polygon constructed by $P_1$, given by
\begin{equation}
P_1^{*} = \left(C_x + r\times \cos\left( \frac{2\pi i}{n}\right), C_y + r\times \sin\left( \frac{2\pi i}{n}\right)\right), 
\end{equation}
for $i = 1, 2, \ldots, n$ and $r = 1$ (radius). 
How to transform $P_1^{*}$ in $P_1$? For example, $P_1 = T \times P_1^{*}$, where $T$ is a transformation matrix.
 A: First, $T$ will have to go on the left, not the right. Second, you need to express $P_1$ in homogeneous coordinates, because otherwise the translation part (i.e., adding $(C_x, C_y)$ to each point) will not be representable by a linear transformation. So rewriting:
\begin{equation}
P_1 = \begin{bmatrix}
x_1 & x_2 & \ldots & x_n \\
y_1 & y_2 & \ldots & y_n \\
1 & 1 & \ldots & 1
\end{bmatrix}
\end{equation}
you can write 
\begin{equation}
Q = \begin{bmatrix}
c & -s &  C_x \\
s & c  &  C_y \\
0 & 0  &  1
\end{bmatrix}\cdot A\cdot B\cdot C
\end{equation}
where $c = \cos (\frac{2\pi}{n}); s = \sin(\frac{2\pi}{n})$, and $A, B,$ and $C$ are the matrices
\begin{equation}
A = \begin{bmatrix}
u & -v & 0 \\
v &  u & 0 \\
0 & 0  &  1
\end{bmatrix};
B = \begin{bmatrix}
\frac{1}{s} & 0 &  0 \\
0 & \frac{1}{s}  &  0\\
0 & 0  &  1
\end{bmatrix};
C = \begin{bmatrix}
1 & 0 &  -C_x \\
0 & 1  &  -C_y \\
0 & 0  &  1
\end{bmatrix}.
\end{equation}
Here $s = \sqrt{(x_1 - C_x)^2 + (y_1 - C_y)^2)}$. 
(BTW, if you decide you want a radius other than $1$, use $r/s$ instead of $1/s$ in the matrix $B$). 
Now all I have to tell you is what $u$ and $v$ are. They are the cosine and sine (respectively) of $-\theta$, where 
$$
\theta = atan2(y_1 - C_y, x_1 - C_x).
$$
A: $$
\newcommand{\bx} {\mathbf x}
\newcommand{\by} {\mathbf y}
\newcommand{\bz} {\mathbf z}
\newcommand{\bw} {\mathbf w}
\newcommand{\bu} {\mathbf u}
\newcommand{\bv} {\mathbf v}
$$
Let $\bx$ denote the column vector with entries $x_1, \ldots, x_n$, and similarly for $\by$, where these are the coordinates of the irregular polygon. Let $\bz$ and $\bw$ denote column vectors filled with the $x$ and $y$ coordinates of the regular polygon. 
Now let 
$$
\bu = \frac{1}{s} \left(\bx - \frac{\bx \cdot \by}{\by \cdot \by} \by \right);\\
\bv = \frac{1}{t} \left(\by - \frac{\bx \cdot \by}{\bx \cdot \bx} \bx \right),
$$
where 
$$
s = (\bx \cdot \bx) - \frac{(\bx \cdot \by)^2}{\by \cdot \by}\\
t = (\by \cdot \by) - \frac{(\bx \cdot \by)^2}{\bx \cdot \bx}.
$$
Note, through direct computation, that $\bu\cdot \bx = 1; \bu \cdot \by = 0; \bv \cdot \bx = 0; \bv \cdot \by = 1$. 
Let $M$ be the $2 \times n$ matrix with rows $\bu$ and $\bv$. And let $K$ be the $n \times 2$ matrix with columns $\bz, \bw$. Then 
$$
KM
$$
takes the vector $\bx$ to the vector $\bz$, and $\by$ to $\bw$. But you wanted it to multiply on the right, with row vectors, so the matrix $T$ you seek is 
$$
M^t K^t
$$
Or at least, that's ONE possible matrix that does what you seek. 
Note that it's essential that the vectors $\bx$ and $\by$ not be parallel for this approach to work. (Indeed, if they are parallel, there's no solution.)
