The condition $0\le y\le a\sin\beta$, $0\le \beta\le\pi/2$ selects a horizontal stripe of width $a\sin\beta$
above the horizontal.
The condition $x\ge y\cot\beta$ means to select a triangular area below the slanted diagonal straight line $x\sin\beta=y\cos\beta$.
The intersection with the horizontal stripe leaves a right-infinite stripe ($x\ge 0$)
with a triangular section at the left.
The line $x\sin\beta=y\cos\beta$ intersects the horizontal
upper rim of the stripe $y=a\sin\beta$ at $x=a\cos\beta$.
The condition $x^2\le a^2+y^2$
selects an area left from a hyperbolic curve opening to the right with apex at $(a,0)$ and
tangent approaching the diagonal $x=y$.
The hyperbola intersects the upper rim of the stripe $y=a\sin\beta$ at $x=a\sqrt{1+\sin^2\beta}$.
Because $\sqrt{1+\sin^2\beta}\ge \cos \beta$, the hyperbola intersects the upper rim at larger $x$,
so this splits the area into a triangular region $0\le x\le a\cos\beta$, $0\le y\le x\tan \beta$,
a rectangular region
$a\cos\beta <x < a$, $0\le y\le a\sin\beta$, and a region delimited by the hyperbola $a<x<\sqrt{1+\sin^2\beta}$, $0\le y\le a \sin\beta$.
As usual we use polar coordinates $x=r\cos\phi$, $y=r\sin\phi$ and Jacobian $r$.
It is easier to split the area into a (i) circular section with a circle of radius $a$ centered at the origin,
which intersects $(a,0)$ at $\phi=0$ as well as $(a\cos\beta, a\sin\beta)$ at $\phi=\beta$,
and (ii) a region outside that circle and left to the hyperbola:
$$
I=\int d\phi\int rdr \ln r^2 = 2\int d\phi r dr \ln r
$$
$$
=2\int_0^\beta d\phi \int_0 r dr \ln r
=2(I_1+I_2).
$$
Circle section:
$$
I_1=\int_0^\beta d\phi \int_0^a r dr \ln r
$$
$$
=\int_0^\beta d\phi [\frac{r^2}{2}\ln r-\frac14 r^2]\mid_{r=0}^a
$$
$$
$$
$$
= \int_0^\beta d\phi[\frac{a^2}{2}\ln a-\frac14 a^2] = \frac{\beta}{2}a^2(\ln a-\frac12).
$$
The OP's solution is apparently missing the contribution from region (ii) or having a sign
error in the formula of the posted problem (ie., using a circle instead of a hyperbola).
In the second region the upper limit of $r$ is given by the intersection of the hyperola
$x^2=a^2+y^2$ and the stripe $y=a\sin\beta$ at $x=a\sqrt{1+\sin^2\beta}$,
which is $r^2=a^2(1+\sin^2\beta)+a^2\sin^2\beta$. $\phi$'s lower limit
is set by the parabola,
$x^2=r^2-y^2=a^2+y^2$, so $y=\sqrt{(r^2-a^2)/2}$, $x=\sqrt{(r^2+a^2)/2}$.
$\phi$'s upper limit is given by the stripe $y=a\sin\beta$, $x=\sqrt{r^2-y^2}=\sqrt{r^2-a^2\sin^2\beta}$,
$\tan\phi=y/x=a\sin\beta/\sqrt{r^2-a^2\sin^2\beta}$,
$$
I_2= \int_a^{a\sqrt{1+2\sin^2\beta}} r dr \ln r
\int_{\arctan[a\sin\beta/\sqrt{r^2-a^2\sin^2\beta}]}^{\arctan[\sqrt{(r^2-a^2)/2}/\sqrt{(r^2+a^2)/2}]} d\phi
$$
$$
= \int_a^{a\sqrt{1+2\sin^2\beta}} r dr \ln r
[
\arctan[\sqrt{(r^2-a^2)/2}/\sqrt{(r^2+a^2)/2}]
-
\arctan[a\sin\beta/\sqrt{r^2-a^2\sin^2\beta}]
]
$$
$$
= \frac12 \int_{a^2}^{a^2(1+2\sin^2\beta)} du \ln \sqrt u
[
\arctan\frac{\sqrt{u-a^2}}{\sqrt{u+a^2}}
-
\arctan\frac{a\sin\beta}{\sqrt{u-a^2\sin^2\beta}}
]
$$
$$
= \frac14 \int_{a^2}^{a^2(1+2\sin^2\beta)} du \ln u
[
\arctan\frac{\sqrt{u-a^2}}{\sqrt{u+a^2}}
-
\arctan\frac{a\sin\beta}{\sqrt{u-a^2\sin^2\beta}}
]
.
$$
By partial integration
$$
K_1=\int du \ln u
\arctan\frac{\sqrt{u-a^2}}{\sqrt{u+a^2}}
$$
$$
=
a^2\ln u \frac{1}{2u\sqrt{u^2-a^4}}
-\int du u(\ln u-1) a^2\ln u \frac{1}{2u\sqrt{u^2-a^4}}
$$
$$
=
a^2\ln u \frac{1}{2u\sqrt{u^2-a^4}}
-\frac{a^2}{2}\int du (\ln u-1) \ln u \frac{1}{\sqrt{u^2-a^4}},
$$
which I cannot solve, so the answer is incomplete at that point.
By partial integration (for $c\equiv a\sin\beta$)
$$
K_2=\int du \ln u
\arctan\frac{c}{\sqrt{u-c^2}}
$$
$$
=
(\ln u-3)
c\sqrt{u-c^2}
+u(\ln u-1)
\arctan\frac{c}{\sqrt{u-c^2}}
+2c^2\arctan\frac{\sqrt{u-c^2}}{c}.
$$