Not really a full answer, but too long for a comment. If I try to numerically integrate your (1) in Mathematica with
a = 0.7; b = 3.1; NIntegrate[Erfc[Sqrt[a/x]] Cos[b x], {x, 0, Infinity}];
(-1/b) Exp[-Sqrt[2 a b]] Sin[Sqrt[2 a b]]
and compare with the 'exact' result, I get agreement (the value is $-0.0350084...$) between the two.
BUT
Mathematica gives me a flag:
NIntegrate::deodiv: DoubleExponentialOscillatory returns a finite integral
estimate, but the integral might be divergent
Such flag is not given when computing the same integral with $\mathrm{erf}$, instead of $\mathrm{erfc}$.
Now, the more I look at it, the more indeed your integral (1) looks divergent to me. There are a number of red flags:
- If you plot the integrand, the oscillations do not 'die out' but persists all the way to infinity. [This does not happen with the $\mathrm{erf}$ version]
- Knowing that $\mathrm{erfc}(z)=1-\mathrm{erf}(z)$, your integral (1) can be effectively written as
$$
\int_{0}^{\infty}\operatorname{erfc}\left(\sqrt{\dfrac ax}\right)\cos(bx)\,dx=
\int_{0}^{\infty}\cos(bx)dx-\int_{0}^{\infty}\operatorname{erf}\left(\sqrt{\dfrac ax}\right)\cos(bx)dx\ ,
$$
i.e. the sum of a divergent integral minus a (probably) convergent one.
- The paper you quote displays a rather suspicious 'asymmetry' (eqs. 4.5.16 and 4.5.17), as you have certainly noted too. If both the integrals with $\mathrm{erf}$ and $\mathrm{erfc}$ were convergent, then I see no reason to omit the other two 'obvious' integrals (where you swap sine with cosine) in the list!
None of this is a smoking gun per se, but my feeling is that (1) is divergent (as well as its sine version), and the paper you quote has a number of typos (except perhaps not the typo I thought it had in a comment I left earlier underneath your question!) - while similar integrals with $\mathrm{erf}$ are convergent (both in their sine and cosine version).
In terms of proving any assertion about such integrals, maybe an approach worth attempting is to write your (1) (or its $\mathrm{erf}$ version) as
$$
\mathrm{Re}\int_{0}^{\infty}\operatorname{erfc}\left(\sqrt{\dfrac ax}\right)\exp(\mathrm{i}bx)\,dx\ ,
$$
and then try integration by parts (the $\mathrm{erfc}$ has a 'nice' derivative, while $\exp(\mathrm{i}bx)$ has a nice antiderivative). This should conclusively prove whether my hunch is correct (no time to do it myself now, though...).