$$I=\int \cos x \left(\frac{\cos (5x)\, d(5x)}{5}\right)$$
$$=(\cos x)\left(\frac{\sin(5x)}{5}\right)-\int (-\sin x)\left(\frac{\sin(5x)}{5}\right)\, dx$$
$$=(\cos x)\left(\frac{\sin(5x)}{5}\right)+\int (\sin x)\left(\frac{\sin(5x)\, d(5x)}{25}\right)$$
$$=(\cos x)\left(\frac{\sin(5x)}{5}\right)+\left((\sin x)\left(\frac{-\cos (5x)}{25}\right)-\int (\cos x)\left(\frac{-\cos (5x)}{25}\right)\, dx\right)$$
$$=(\cos x)\left(\frac{\sin(5x)}{5}\right)+(\sin x)\left(\frac{-\cos (5x)}{25}\right)+\frac{I}{25}$$
Solve for $I$.
$$I=\frac{25}{24}\left((\cos x)\left(\frac{\sin(5x)}{5}\right)+(\sin x)\left(\frac{-\cos (5x)}{25}\right)\right)+C$$
$$=\frac{5}{24}\cos x\sin(5x)-\frac{1}{24}\sin x\cos (5x)+C$$
This agrees with your book's answer.
Without using integration by parts: use the identity $$\cos x\cos y=\frac{1}{2}(\cos(x+y)+\cos(x-y))$$
$$\int \cos x\cos (5x)\,dx =\frac{1}{2}\int (\cos(6x)+\cos (4x))\, dx$$
$$=\frac{1}{2}\left(\frac{1}{6}\int \cos (6x)\, d(6x)+\frac{1}{4}\int \cos(4x)\, d(4x)\right)$$
$$=\frac{1}{12}\sin(6x)+\frac{1}{8}\sin(4x)+C$$