# Enforce certain orientation for a local coordinate system.

This problem requires some explanation, maybe because it's very special or I just don't know the right terminology. In any way, please bear with me as I try to explain this as best as I can.

Given are the five vectors $$a$$, $$b$$, $$c$$, $$n$$ and $$r$$. The vectors $$a$$, $$b$$ and $$c$$ each describe points on the same plane.

I want to define a local coordinate system by using the plane defined by the three vectors $$a$$, $$b$$ and $$c$$ for the $$X/Y$$-plane of the coordinate system and the normal of that plane as the $$Z$$-axis. The origin of that coordinate system should lie in the point described by $$n$$.

I construct the vectors $$x$$, $$y$$ and $$z$$, that define that coordinate system, as follows (note that they'll each have to be converted to unit-vectors as well to keep the scale of the global coordinate system, I've omitted that step here for simplicities sake):

• $$z = (a - b) \times (c - b)$$
• Let $$u$$ be the vector $$(0,1,0)$$ of the global coordinate system (the global "up")
• $$x = z \times u$$
• $$y = x \times z$$

Now this is where I run into a problem. This coordinate system works as is, but each axis-defining-vector could also be its inverse. To prevent ambiguity the local coordinate system should also fulfill the following criteria (where $$r$$ defines a point that isn't on the plane defined by $$a$$, $$b$$ and $$c$$):

1. Given an observer in $$r$$, the $$z$$ of the local coordinate system should point out of the plane towards the observer
2. The $$y$$ of the local coordinate system should point in the direction, where it is "closest" to the global "up". That is, given both alternatives for $$y$$ you should choose the one pointing "upwards".
3. Given that same observer in $$r$$ as in criteria 1, the $$x$$ should point to the right if the observer were to look at the plane.

I have no idea how I can check if my calculated $$x$$, $$y$$ and $$z$$ fulfill these criteria and therefore I also don't know if I need to invert them. I'd be thankful for any idea in this regard as I've really struggled to find a solution for this.

All of this is easily accomplished via dot and cross products. Assuming that $$a$$, $$b$$, $$c$$ and $$n$$ are coplanar and that $$r$$ does not lie on this plane,
1. Examine the sign of $$z\cdot(r-n)$$. If it is positive, then $$r$$ is in the half-space into which $$z$$ points. If negative, it’s on the opposite side of the plane, so flip $$z$$. In other words, to point $$z$$ in the desired direction, multiply $$z$$ by the sign of $$z\cdot(r-n)$$ regardless.
2. Similarly, multiply $$y$$ by the sign of $$y\cdot u$$, i.e., by the sign of its own second coordinate. If that’s zero, $$y\perp u$$ and neither $$y$$ nor $$-y$$ is “closer” to pointing upwards.
3. Here, I’ll assume that this observer is oriented with her “up” aligned with global “up.” Once you’ve aligned $$y$$ and $$z$$ correctly, this criterion is satisfied by having $$x$$ be clockwise of $$y$$ when looking toward the plane from $$r$$, so set $$x=y\times z$$, normalized, of course. Again, if $$y\perp u$$, $$x$$ will end up pointing “upward” or “downward,” but this will at least ensure that the $$x$$-direction is in some sense to the right of the $$y$$-direction in that case.