Matching two coordinate systems with three vectors

This may be a bit long winded but I have not done linear algebra in a while and I want to make sure my math is correct.

I am running two different computer simulations of proteins and I need to match them by mapping the peptide bond of one simulation onto another. For those who do not know much chemistry it looks like this:

All that really matters though is that I have three vectors all within one plane in both simulations. One from the C atom to the O atom, one from the C atom to the N atom, and one from the N atom to the H atom(white ball). I also have two different position vectors for a point, one for each simulation, P1 and P2, with the origin of the vector at C.

I will call these vectors, $\vec{CO}$, $\vec{CN}$ and $\vec{NH}$. Since there are two simulations both with these vectors, the other set will be $\vec{CO'}$, $\vec{CN'}$, and $\vec{NH'}$.

What I am doing is finding the cross product $\vec{CO}\times{\vec{CN}}$, calling it $\vec{w}$, and then finding the cross product $\vec{w}\times{\vec{CO}}$, and then normalizing all these vectors and putting them in a matrix $F$, and doing the same thing for the other set and calling it $H$. Then I am finding a transformation matrix by finding dot product of $H$ and the inverse of $F$.

$$A\cdot{F}=H$$ $$A=H\cdot{F^{-1}}$$

Once I have the transformation matrix $\vec{A}$ I am applying into to $\vec{P1}$ like this:

$$A\cdot{\vec{P1}}=\vec{P1'}$$

and then measuring the distance between $\vec{P1'}$ and $\vec{P2}$. The problem I am having is that $\vec{P1'}$ is varying way too much between every step of the simulation. This is leading me to believe that I have done something wrong with the transformation matrix. Any help is appreciated and thank you in advance!

• £I see nothing wrong in this derivation. Did you check that the matrices $F$ and $H$ are indeed orthogonal (so that you can use $F^{-1}=F^T$) ? Did you check that you made the product in the right order ? – Yves Daoust Aug 9 '18 at 16:06
• I had not, but I just checked now and they are indeed orthogonal. And they do seem to be in order. – John Kent Aug 9 '18 at 16:14
• You can also apply the matrices to the known vectors $CO, CN$ and check that you get $(x, 0, 0)$ and $(x, y, 0)$ coordinates. Then check the effect of $A$ on these vectors. – Yves Daoust Aug 9 '18 at 16:35
• Do your simulations use periodic boundary conditions? If so, have you checked that all your vectors between pairs of atoms within the peptide group have had a minimum image correction applied? Almost certainly this won't be the answer to your problem, but it is something that occasionally gets forgotten. – user575517 Aug 9 '18 at 16:41
• Is the C atom at the origin in both systems? If not, perhaps you also need to adjust the origin when you transform from one system to the other. An example of the phenomenon you’re trying to get rid of would be helpful. – amd Aug 9 '18 at 17:46