I have a 3D polygon based on a set of faces. Each face lies in a single plane, and I know the normal for each face (as well as the points that create the face). I also know that each normal points "outward".
I need to figure out which faces contribute to the polygon being concave (if any).
For some reason I'm having a hard time conceiving this...
I first thought I might be able to simply check the dot-product of each neighbouring normal, and simply see which ones are < 0. But I guess this method would fail because the dot product between normals doesn't share information about whether two faces "face" eachother or not. In fact, I think I could even think of cases where the dot product of the normals is not consistent (i.e. some dot products >0 and some <0), and the polygon would still be convex.
For the 2D case I guess I could traverse the polygon in the order it was defined and just check the angles from one face to the next. In this case, the edges that contribute to the polygon being concave would be the faces that create an angle of larger than 180 degrees with their previous face.
I'm not even sure if this would even work in a 3D case.