For any given $h$ and $\theta$, there is a critical radius $R=R_0$
at which the sphere is tangent not only to the curved side of the cone
but also to the surface of the water at the top of the cone.
The sphere with radius $R=R_0$ displaces more water than any smaller
sphere, so clearly the displacement is maximized for some
$R \geq R_0.$
There is a larger spherical radius, $R=R_1$, at which the sphere is
tangent to the cone but the circle of tangency is exactly at the
surface of the water. If $R > R_1$ then we have a sphere that touches
the cone only at the top edge of the cone; the part of the sphere
inside the cone is a subset of the part of the sphere of radius $R=R_1$
inside the cone, so the volume of water displaced is maximized for some
The volume of water displaced is then the volume of the sphere
minus the volume of whatever part of the sphere is above the
plane of the surface of the water.
If the highest point on the sphere is at a distance $d$
above that plane, the part of the sphere above that plane is
a spherical cap
of height $d$ from a sphere of radius $R,$ which has a volume
\frac13 \pi d^2(3R-d).
Possibly you were meant to work out that formula for yourself
using a disk method of integration.
Alternatively, let $d$ be the distance from the lowest point of the sphere
to the plane at the top of the cone, and then the volume of water displaced
is the volume of a cap of height $d$.
(Either formula works just as well for $R < d \leq 2R$ as for $0\leq d\leq R.$)
In any case,
you can assume $R_0 \leq R \leq R_1$, that is, the sphere is tangent to
the curved surface of the cone and also intersects the plane at the
top of the cone.
Then $d$ is a function of $h,$ $\theta,$ and $R.$
Treating $h$ and $\theta$ as constants
and $R$ as a variable, the volume of the part of the sphere
within the cone is a function of $R$ that you can maximize.
To complete the solution, you should confirm that the answer is
in the range where your formulas are valid, that is,
$R_0 \leq R \leq R_1.$