I have two images. One is the reference image and the other is the rotated version of the reference image. I would like to find what is the angle by which the second image is rotated with respect to the first one. i want this to work automatic.
Enumerate over all possible angles (under some reasonable discretization), rotate one of the images, and calculate the $L_2$ distance (sum of squared differences); the angle at which the distance is minimal is the one you seek.
An even better idea is to calculate the two-dimensional Fourier transform, and now do the same with shifts instead of rotations (rotation in the time domain corresponds to shift in the frequency domain). Maybe you can reduce the amount of shifts tested by looking at some prominent peeks of the spectrum and trying to match them.
I have no idea how to implement these suggestions in Matlab, but I'm sure you can find out.
Both solutions assume that the image is "continuous" with reasonable "modulus of continuity". If the image is random that the methods wouldn't work.
You can look at optical flow, OpenCV has a function that provides this functionality.
Optical flow returns points that are similar between two successive frames, which can be used to calculate a rotation. This function will work best on twoframes that are not too different from eachother. I am not sure how big a rotation we are talking about here, if it is too high, optflow might not be the best option.
As mentioned in the link above, using SIFT might a better option. SIFT calculates features that can be used to match objects, corners from different angles, even with occulusions (so you only see part of the object, but SIFT numbers can still match), in that case, if there are enough SIFT matches between two features, these can be used to calculate a rotation.