Calculation Projectionmatrix of a Camera by XYZ - UV Pairs

i have broken down my problem to plainmath and could really use some help.

Basis: I have an image. In this image I have several UV-XYZ pairs. So i know the 3d position of serveral Pixels.

Given the following equations from 3d to 2d space.

$Z_{c} \begin{pmatrix} U \\ V \\ 1 \end{pmatrix} = \begin{pmatrix} f & 0 & px \\ 0 & f & py \\ 0 & 0 & 1 \end{pmatrix} \begin{pmatrix} r_{1} & r_{2} & r_{3} & t_{1}\\ r_{4} & r_{5} & r_{6} & t_{2}\\ r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix} \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix}$

f, px, py are known constants. Many $UV \to XYZ$ pairs are known. I want to get $r_{1} - r_{9}, t_{1}, t_{2}$ and $t_{3}$

I can transform to:

$\begin{pmatrix} f & 0 & px \\ 0 & f & py \\ 0 & 0 & 1 \end{pmatrix} ^{-1} \begin{pmatrix} U \\ V \\ 1 \end{pmatrix} = \frac 1 Z_{c} \begin{pmatrix} r_{1} & r_{2} & r_{3} & t_{1}\\ r_{4} & r_{5} & r_{6} & t_{2}\\ r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix} \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix}$

The $Z_{c}$ is the Z-Coordinate of the point AFTER the multiplication with the unknown transformation matrix.

$Z_{c} = \begin{pmatrix} r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix} \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix}$ This leads to these equations for U and V: $U_{k} = \frac {r_{1}X + r_{2}Y + r_{3}Z + t_{1}} {r_{7}X + r_{8}Y + r_{9}Z + t_{3}} V_{k} = \frac {r_{4}X + r_{5}Y + r_{6}Z + t_{2}} {r_{7}X + r_{8}Y + r_{9}Z + t_{3}}$

The question in the end is quite simple. I need to transform this equation into sth., so I can apply 50-100 $UV \to XYZ$ pairs and get the camera position and rotation. Without the $Z_{c}$ it is pretty easy to transform into sth. like this:

$\begin{pmatrix} U_{1} + V_{2} + 1 \\ U_{4} + V_{5} + 1 \\ \vdots \end{pmatrix} = \begin{pmatrix} X_{1} & Y_{1} & Z_{1} & 1 & X_{2} & Y_{2} & Z_{2} & 1 & X_ {3} & Y_{3} & Z_{3} & 1 \\ X_{4} & Y_{4} & Z_{4} & 1 & X_{5} & Y_{5} & Z_{5} & 1 & X_ {6} & Y_{6} & Z_{6} & 1 \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \end{pmatrix} \begin{pmatrix} r_{1} \\ r_{2} \\ r_{3} \\ t_{1} \\ r_{4} \\ r_{5} \\ r_{6} \\ t_{2} \\ r_{7} \\ r_{8} \\ r_{9} \\ t_{3} \end{pmatrix}$

I hope you get the idea. I dont know if its correct, but it seems like it. So this is a classical overdetermined linear equation (if you add 100 Rows), which I could have solved with a QR-Decomposition (i hope its called like that). But I can't apply this idea to the new Problem with $Z_{c}$. Leaves me clueless.

-