Consider the following diagram:
The "2D cone" with origin A separates the 2D plane into 2 regions, inside and outside.
Consider the 4 representative edges $c, d, e, f$ that encompass all intersection cases (assume that an edge aligned with the cone's boudary is equivalent to $e$).
An edge is either fully inside the cone, has one end point inside and one outside, has both end points outside but a subset of its interior is inside, is fully outside.
We are looking for the points of intersection of the cone $A$ with each edge. For $c$ it would be the rightmost end point and the point of intersection of the cone's edge with the segment, for $e$ it would be the 2 endpoints of the segment, for $f$ no intersection point exists, for $d$ it's the 2 points where the boundary of the cone intersects the interior of $d$.
With this setup here is the practical problem:
The cone $A$ is defined by an origin $A$ and 2 directions $d_1, d_2$. Each segment is defined by its 2 end points $p_1, p_2$.
Given an arbitrary cone, with inner angle less than $\pi$ and an arbitrary segment, use nothing but vector algebra to find the 2 intersection points. If no intersection is possible, identify it somehow, encoded numerically in the 2 points. You can assume the directions $d_1, d_2$ are always given to you in clockwise order.
My current approach is to, grab the 2 end points of the segment, check their angled sign relative to the window (which is done through the dot product of 2 cross products). With the signed angles of each end point I identify whether the point is inside or outside the window. A point is in the interior iff $0 < \sigma < w$ where $\sigma, w$ are the signed angle of the point with respect to the rightmost edge of the cone and $w$ is the angle of the cone.
With that information I can decide which of the 4 cases I am actually in, then make the decisions with that assumption.
e.g If only one of the 2 endpoints is in the interior I know for a fact there is a unique point of intersection with one of the 2 boundaries, so figure out which one it is and then I know the 2 intersection points.
This is overly convoluted. I am curious if there is a more unified way that can find both intersection points without having to create a big tree of