whenever we encounter fractions with a complex number in the denominator, we multiply both parts of the fraction with the complex conjugate of the denominator to make the denominator a real number. (for $z=x+yi$, the conjugate is simply $\overline{z}=x-yi$). Multiplying with the conjugate both places will result in:
$\frac{(3+ai)(5-bi)}{(5+bi)(5-bi)}=\frac{(15+ab)+(5a-3b)i}{25-b^2}$
if you expand the fraction you will more easily see the difference in the real and imaginary part.
$\frac{15+ab}{25-b^2} + \frac{5a-3b}{25-b^2}i$
Then, as the excercise states, the real part is 3 and the imaginary part is 4. This is same same as saying
$\frac{15+ab}{25-b^2}= 3$ and $\frac{5a-3b}{25-b^2}=4$
From there, you can just solve those two equations for a and b yourself or use some CAS-tool.