You can treat this integral as an integral over a region of the Cartesian plane rather than as one single-parameter integral inside another, and it is useful to do so.
To check the work you've done so far, let's consider the case of your integral when $a=b=1$. Then the region of integration is the unit square. If $f$ is even, then every region below the diagonal is matched by a symmetric region above the diagonal, so we can take twice the integral below the diagonal to get the integral over the square. This then agrees perfectly with what you already found, although it depends on the function being even and (as you noted) it leaves an inconvenient region still to be integrated if $a\neq b$.
The following method can be adapted to integrate your leftover region, but I think that is as much work as just applying it to the entire original region of integration, so I will do the entire region.
Treating your double integral as an integral over a region in the plane, you want to integrate $f(x-y)$ over the rectangle with vertices at $(0,0)$, $(0,a)$, $(b,a)$, and $(b,0)$.
Transforming the plane according to the rule $(x,y) \mapsto (x-y, y)$, your original integral is equal to the integral of $f(x)$ over the parallelogram with vertices at $(0,0)$, $(-a,a)$, $(b-a,a)$, and $(b,0)$.
You can rewrite that as three integrals over functions of $x$ alone, or if you like, as one integral over a function of $x$ defined piecewise using $f(x)$.
One of the three integrals comes from integrating $f(x)$ over the region of integration to the left of the $y$ axis.
This region is the triangle with vertices at $(0,0)$, $(-a,a)$, and $(0,a)$, its cross-section along the line $x=c$ is $c+a$,
and the integral is
This is equal to the integral
from your calculations, which of course it must be since it's the integral of $f(x-y)$ over the part of the original rectangular region of integration above the line $y=x$.
The other two integrals over $f(x)$ are the integral over the rectangular subregion between $x=0$ and $x=b-a$ and the third is the triangle to the right of the line $x=b-a$.
If $f$ is even, you can assume $b>a$ without loss of generality (although it appears you were already aware of that). This makes the procedure above a little easier than in the more general case ($f$ not even) since you do not have to be concerned about the regions to use if $a<b$.