Why do we find the volume of a prism this way with vectors? Take the following prism:

To find its volume we first find the vector product of $\vec{OQ}$ and $\vec{OP}$:
$$\begin{bmatrix}i & j & k\\4 & -5 & 1\\3 & -3 & 1\end{bmatrix} = -2i -j + 3k$$
Then you find $\vec{QT}$:
$$\begin{bmatrix}4 \\-4 \\16 \end{bmatrix} - \begin{bmatrix}3 \\-3 \\1 \end{bmatrix} = \begin{bmatrix}1 \\-1 \\15 \end{bmatrix}$$
From there we perform a dot product operation:
$$\vec{QT}.\vec{OP}*\vec{OQ} = 44$$
Then $44 / 2 = 22$ so the volume is $22 units^3$.
I follow this up until the dot product is used.  My initial instinct when tackling this problem was to find the area of the triangle $OQP$ by performing $\frac{1}{2}|OP * OQ|$, and then finding the length of the line $QT$ by calculating $\sqrt{(x_1 - x_2)^2 + (y_1 - y_2)^2 + (z_1 - z_2)^2}$, and then multiplying the area of the triangle $OQP$ by the length of line $QT$, but this gave me $28.2units^3$.
1) How/Why does the dot product find the volume of the cuboid (before you half it to get the prism volume)?
2) Why did my instinctual method not work?
 A: You already know that the area of the triangular base is $\frac12\|\vec{OP}\times\vec{OQ}\|$, so I’ll skip that part of the explanation, but I’ll remind you that that cross product is the (signed) area of the parallelogram with sides defined by the two vectors.  
The volume of the prism is the area of its base times its height. That height is $\|\vec{QT}\|\cos\phi$, where $\phi$ is the angle that $\vec{QT}$ makes with a perpendicular to the base. As has been point out out already, your method would’ve worked if $\phi$ were 90°, but in this case it isn’t. Recall now that the dot product of $\vec a$ and $\vec b$ is $\|a\|\,\|b\|\cos\phi$, where $\phi$ is the angle between them. If we can find a unit vector perpendicular to the base, we can then use a dot product to get the height of the prism. The cross product $\vec n=\vec{OP}\times\vec{OQ}$ is perpendicular to both $\vec{OP}$ and $\vec{OQ}$ and so is perpendicular to the base, therefore the height of the prism is $$\vec{QT}\cdot{\vec n\over\|\vec n\|}$$ (we’ve taken care to take the cross product in the right order so that $\vec n$ is on the same side of the base as $\vec{QT}$). We then multiply this by the area of the base to get $$\frac12\|\vec{OP}\times\vec{OQ}\|\left(\vec{QT}\cdot{\vec n\over\|\vec n\|}\right)=\frac12\vec{QT}\cdot(\vec{OP}\times\vec{OQ}).$$. 
The quantity $\vec a\cdot(\vec b\times\vec c)$ is known as the scalar triple product, and just as the length of the cross product of two vectors gives the area of the parallelogram that they span, the triple product gives the volume of the parallelepiped spanned by the three vectors. If you write the product out in terms of the coordinates of the vectors, after some manipulation you’ll find that it’s equal to the determinant $$\begin{vmatrix}x_a&y_a&z_a\\x_b&y_b&z_b\\x_c&y_c&z_c\end{vmatrix}.$$ This form of the product should make the connection to the two-dimensional area more apparent, since that can also be written in terms of the vector coordinates as $$\begin{vmatrix}x_a&y_a\\x_b&y_b\end{vmatrix}.$$ This interpretation of determinants generalizes to higher dimensions.
A: Your "instinctual" method would work if the line $QT$ was perpendicular to the plane $OQP$. But it isn't.
For the principle of the cuboid's volume, one can appeal to Cavalieri's principle. To quote Wikipedia

Suppose two regions in three-space (solids) are included between two
  parallel planes. If every plane parallel to these two planes
  intersects both regions in cross-sections of equal area, then the two
  regions have equal volumes.

The parallel planes here are $OPQ$ and $RST$. We compare the prism with
a cuboid with a cuboid where $R$ is replaced by $R'$ with $R'$ on
$RST$ and $OR'$ perpendicular to $OPQ$. We also replace $S$ and $T$ by $S'$ and $T'$. By Cavalieri, the two prisms have the same volume. But the
latter volume is area($OPQ$)$|OR'|$. This is half the dot product of the vector $OR$ and the cross product of vectors $OP$ and $OQ$.
