Finding whether a line intersects with a triangle I am writing a 3D program (with OpenGL) and need to do ray-tracing to find if the user has clicked on a model. I have the coordinates of the vertices (all triangles) in 3d space and the coordinates of the line I am using for the ray-tracing. So far, the best formula I have come up with is this:
P+rV=Ax+By+C(1-x-y)
where the left side of the equation is the formula for points on a line, and the right side is the formula for points on a triangle, the idea behind it being that if there is a solution to the equation, then there is a collision.
I've solved some equations like these in my linear algebra class in college, but nothing this complicated, so any help is appreciated.
edit: P is the origin of the line, r is a scalar value, V is the direction of the line, A, B, and C are the vertices of the triangle, and x and y are scalars.
 A: The first thing to do is to find the intersection of the line with the plane of the triangle. The plane can be found by first computing the cross product of $\mathbf{b-a}$ and $\mathbf{c-a}$ to get a normal $\mathbf n$ to the plane, then finding $k=\mathbf{n\cdot a}$; the plane is the set of vectors $\mathbf w$ with $\mathbf{n\cdot w}=k$. Intersecting the line with the plane then means solving
$$\mathbf n\cdot(\mathbf p+r\mathbf v)=k$$
which is easy because it is a linear equation in $r$; call the solution $r^*$. The point of intersection is $\mathbf x=\mathbf p+r^*\mathbf v$. We can now apply any method for determining if $\mathbf x$ lies in the triangle, and that will solve our line-triangle intersection; Stack Overflow gives several algorithms.
A: By $Ax+By+C(1-x-y)$ you try to make the point to belong to the triangle as I understand. Then it should be a convex combination of $A$, $B$, $C$, that is $x$, $y$ and $1-x-y$ have to be non-negative (barycentric coordinates). Let's call $z=1-x-y$, then you have to solve the linear system (4 equations and 4 unknowns)
$$
\begin{align}
Ax+By+Cz+Vr&=P,\\
x+y+z&=1.
\end{align}
$$
If it is solvable and $x\ge 0$, $y\ge 0$, $z\ge 0$ then the line goes through the triangle, otherwise it is not.
