There is a calculation error in this that I haven't found, but the idea will work. The main tools that I will use consist of elements of linear algebra and multivariate calculus. Many of these calculations can be done in two ways and I'll pick whichever one I like best. For programming situations, some of these operations are less stable than others, so you might need to replace some of these steps with other options.
Let $v_1=(0,5,0)$, $v_2=(0,0,0)$, and $v_3=(8,0,0)$ be the vertices of triangle $O$. Let $w_1=(6,8,3)$, $w_2=(6,8,-2)$, and $w_3=(6,-4,-2)$ be the vertices of triangle $Y$.
Step 1: We begin by determining the planes that contain these triangles. The normal to the plane containing $O$ is given by
$$
(v_2-v_1)\times(v_3-v_1)=(0,5,0)\times(8,0,0)=(0,0,-40).
$$
Therefore, the plane containing $O$ is
$$
0(x-0)+0(y-0)-40(z-0)=0.
$$
In other words, $z=0$.
The normal to the plane containing $Y$ is given by
$$
(w_2-w_1)\times(w_3-w_1)=(0,0,-5)\times(0,-12,-5)=(-60,0,0).
$$
Therefore, the plane containing $Y$ is
$$
-60(x-6)+0(y-8)+0(z-3)=0.
$$
In other words, $x=6$.
Step 2: Now, let's determine the line of intersection of these planes. The line must satisfy $z=0$ and $x=6$, so, as an augmented matrix, we have
$$
\begin{bmatrix}
1&0&0&6\\0&0&1&0
\end{bmatrix}
$$
This matrix is already in reduced row echelon form with $x=6$, $y$ a free variable, and $z=0$. Therefore, the line of intersection is $(6,t,0)$.
Computing the rref is not always a numerically stable operation, so you might want to replace this step with taking the cross product of the normal vectors of the planes to get the direction of the line of intersection and finding a nice point on the line of intersection.
Step 3: Find the intersection of the line and triangle $O$. The line containing $v_1$ and $v_2$ has direction $v_2-v_1=(0,5,0)$ and passes through the point $(0,0,0)$, so its formula is
$$
(0+0s,0+5s,0+0s)=(0,5s,0).
$$
(I changed variables because I already used $t$ in the previous formula). The other sides of the triangle are given by the lines
$$
(0+8s,0+0s,0+0s)=(8s,0,0)
$$
and
$$
(8-8s,5s,0)=(8-8s,5s,0).
$$
(Be careful about setting this up, you really want to write $sv_1+(1-s)v_2$ so that when $s=0$, you get one of the endpoints and when $s=1$, you get the other endpoint.)
To find intersection points, we must solve three systems of equations. The first is to intersect the line of intersection with the first side of the triangle. The first system of equations is
$$
6=0\qquad t=5s\qquad 0=0
$$
The matrix form is
$$
\begin{bmatrix}
0&0&6\\
1&-5&0\\
0&0&0
\end{bmatrix}
$$
This system has no solution, so there is no intersection point.
The second system is
$$
6=8s\qquad t=0\qquad 0=0
$$
The matrix form is
$$
\begin{bmatrix}
0&8&6\\
1&0&0\\
0&0&0
\end{bmatrix}
$$
The solution to this system is $t=0$ and $s=\frac{6}{8}=\frac{3}{4}$. Since the $s$-value is between $0$ and $1$, this point lies on the side of the triangle. This point is $(6,0,0)$.
The third system is
$$
6=8-8s\qquad t=5s\qquad 0=0
$$
The corresponding matrix is
$$
\begin{bmatrix}
0&8&2\\
1&5&0\\
0&0&0\\
\end{bmatrix}
$$
This system has a solution, $s=\frac{1}{4}$ and $t=\frac{5}{4}$. Therefore, the point of intersection is $(6,\frac{5}{4},0)$.
Therefore, the segment on the first triangle is between $(6,0,0)$ and $(6,\frac{5}{4},0)$. This is between $t=0$ and $t=\frac{5}{4}$.
For triangle $Y$, I won't show all the work, but give the highlights. The lines for the sides of the triangles are given by $(6,8,3-5s)$, $(6,8-12s,3-5s)$, and $(6,8-12s,-2)$.
The first line intersects at $t=8$ and $s=\frac{3}{5}$, which corresponds to the point $(6,8,0)$. The second line intersects at $t=\frac{4}{5}$ and $s=\frac{3}{5}$, which corresponds to the point $(6,\frac{4}{5},0)$. The third line does not intersect the intersections of the planes.
Now, the first triangle intersects the line of interest between $t=0$ and $t=\frac{5}{4}$ while the second triangle intersects between $t=\frac{4}{5}$ and $t=8$. Intersecting these two segments gives the intersection between $t=\frac{4}{5}$ and $t=\frac{5}{4}$ (this isn't right, I'll search for the error).
