It is defined that:
\begin{align} O(\omega)&=\frac{1}{\sqrt{2\pi}}\int O(t)e^{-i\omega t} \mathrm{d}t \tag{1} \\ O^{\dagger}(\omega)&=\frac{1}{\sqrt{2\pi}}\int O^{\dagger}(t)e^{-i\omega t} \mathrm{d}t \tag{2} \end{align} If I directly take the hermitian conjugate of equation (1) then I get: \begin{equation} O^{\dagger}(\omega)=\frac{1}{\sqrt{2\pi}}\int O^{\dagger}(t)e^{i\omega t} \mathrm{d}t \end{equation} This result is different from equation (2). Why?
The Fourier transform of the conjugate of a function is indeed not equal to the conjugate of the Fourier transform. Complex conjugation and Fourier transformation do not commute.
You essentially proved this, maybe it's clearer if we use a less ambiguous notation. Let $$ \hat{f}(\omega)=\frac{1}{\sqrt{2\pi}}\int_\mathbb{R} f(x) \mathrm{e}^{-i \omega x}\mathrm{d}x $$ $$ \hat{\overline{f}}(\omega)=\frac{1}{\sqrt{2\pi}}\int_\mathbb{R} \overline{f(x)} \mathrm{e}^{-i \omega x}\mathrm{d}x $$ As you point out, taking the conjugate of the first we obtain $$ \overline{\hat{f}(\omega)}=\frac{1}{\sqrt{2\pi}}\int_\mathbb{R} \overline{f(x)} \mathrm{e}^{i \omega x}\mathrm{d}x =\hat{\overline{f}}(-\omega) \ . $$
This is not so surprising if you think for example that the FT of a real function is not in general real.
