Convert a pixel displacement to angular rotation? I have camera at which i know the distance to it, its HFOV, and each frame has resolution axb. 
How would one convert a pixel displacement between the center of frame and the object into a propper angular movement. 
The way i would do it HFOV/a * pixel displacement.. but that just feel incorrect?
 A: The formula you gave does not work (it assumes linear dependence on angle).  I will describe how you generate a formula that will give you the angle you're looking for.
For the moment, assume we are only working with the $x$-axis (the argument is the same for the $y$).  We have $b$ pixels in the $x$ direction.  Imagine a line drawn from the camera to the center of the image (it has length $\ell$ that will turn out not to really matter).  We can draw a right triangle whose hypotenuse goes from the camera to the edge of the image.  That angle made between the base (of length $\ell$ which is perpendicular to the image and meets at the center) and the hypotenuse is $\theta$ ($\theta=\frac{1}{2}HFOV$).  Suppose that we are interested in the pixel that is $x$ pixels away from center.  Draw a line from the camera to this point and call the angle it makes with the base $\phi$.  Angle $\phi$ is the one you are interested in finding.
Using some right-triangle trig, we have two triangles with a common base.  We have that $\tan(\theta)=\frac{b/2}{\ell}$ and $\tan(\phi)=\frac{x}{\ell}$.  Since $\ell$ is the same in both equations, we solve both for $\ell$, namely: $\ell=\frac{b/2}{\tan(\theta)}$ and $\ell=\frac{x}{\tan(\phi)}$.  Since the $\ell$ is unchanging, we have that $\frac{b/2}{\tan(\theta)}=\frac{x}{\tan(\phi)}$.  You know $b$ (width in pixels of image), $x$ (number of pixels from center), and $\theta$ (HFOV/2).  If you want to know $\phi$ you solve the equation for $\phi$:
$$\phi = \tan^{-1}\left(\frac{x\tan(\theta)}{b/2}\right)=\tan^{-1}\left(\frac{2x\tan(\theta)}{b}\right).$$
