I'm working on a tool for game modding, intended to help users create mesh morphs. This part of the tool concerns joint-based morphs.
My problem is this: I want users to be able to enter delta translation, rotation, and scaling in either local or world coordinates, and if world coordinates are entered have the tool convert them into the joint's local coordinates which the game requires. I've got it working except for scale.
I have the world transform matrix, a rotation-only world 3x3 matrix, and a world quaternion available. I can multiply world rotation matrix * the local scale-only matrix and extract the world scale from the resulting matrix no problem. I can then multiply the world rotation matrix inverse * the world rotation/scale matrix I created in the previous step and get my local scale back. I can multiply the world rotation matrix inverse * a world scale matrix and get the correct local scale if the rotation is relatively simple as in 90 degrees around one or two axes.
The problem is that I need to be able to change the world-space scale and calculate the local scale from it for more complex rotations as in the model's hands. With these complex rotations the conversion not only doesn't recover the original local scale, the values keep changing as I flip back and forth between local and world.
In other words:
world scale = extracted from world rotation matrix * local scale matrix
change the world scale
local scale = extract from world rotation matrix inverse * ?
It seems to me that I need a world rotation matrix with the world scale added without being rotated, but I can't seem to figure out how to do that.
Here's the sample rotation matrix I'm working with:
0.707404566002685, -0.705833615457916, 0.0371181951744841, -0.704524320694, -0.708363939102517, -0.0431981886573919, 0.0567839374145007, 0.00440792017033548, -0.998376680063813
Some of the things I've tried:
world rotation matrix inverse * world scale matrix
world rotation matrix transpose * world scale matrix
world rotation matrix inverse * (world scale matrix * world rotation matrix)
world rotation matrix inverse * world rotation matrix with world scale added directly to it by multiplying the diagonal by the x, y, z scale values
Nothing gives me a round trip back to the original scale without using the world scale matrix from the first step.
Disclaimer: I'm not a mathematician so please forgive my ignorance and keep answers as simple as possible.
Edit: After more testing I see the local rotation I get from multiplying the rotation matrix inverse * the world scale seems to give me the same modification of the mesh as the original local scale despite being different values. Does this make sense?