I'm designing an Extended Kalman Filter which will take several types of measurements and try to estimate a location. One type of measurement is the direction to the location.
I've thought about this a fair bit, and it seems to me that using the direction as a measurement itself is prone to problems; either the $0/2\pi$ problem if it's given as an angle, or if it is given as a unit vector there will be too many degrees of freedom when the innovation is calculated.
It seems to me that the "right" thing to do is to pretend the "measurement" $z$ is always $1$, and use the cosine between the direction vector and the vector to the current estimate as the Kalman observation function $h$.
It also requires that the measurement noise be transformed to the cosine domain (presumably $HPH^T$ for the transformation jacobian $H$), where it's not clear it would still be Gaussian.
Basically, this approach would require me to compute the Jacobian of the cosine calculation two different ways: first holding the state estimate $x$ constant (to transform the measurement covariance), then holding the measurement $z$ constant (to compute the EKF observation matrix Jacobian). I'm worried about the mathematical soundness of this approach.
Any suggestions or insights?