Do we ever study "mixed" categories? Consider a category whose object class includes the class of all topological spaces and the class of all topological groups. Furthermore, let the hom-sets between any two objects be the usual hom-sets in the category of topological spaces, unless both objects are topological groups, in which case the hom-set between them will be the usual hom-set in the category of topological groups.
Are "mixed" categories such as this ever studied?
 A: What you've described is not a category, since it is not closed under composition of morphisms (since if you consider group $\to$ space $\to$ group, you allow the two arrows to be any continuous maps, and then there is little chance that there composite will be a group homomorphism).
A: Yes:
if categories $\Bbb A$ and $\Bbb B$ are disjointly contained in a (bigger) category $\Bbb H$ as full subcategories, and suppose that $\Bbb H$ has no other objects, then I call $\Bbb H$ a bridge between $\Bbb A$ and $\Bbb B$, see my paper.
As Matt E. answered, your particular example is not a bridge, because we allow both directions $\Bbb A\not\to\Bbb B$ and $\Bbb B\not\to\Bbb A$. However, if we keep only one direction, say $\Bbb A\not\to\Bbb B\ $ (letting the homsets $\hom(b,a):=\emptyset$), we do get a category.
Such directed bridges from $\Bbb A$ to $\Bbb B$ straightly correspond to profunctors, i.e. functors of the form $\Bbb A^{op}\times\Bbb B\to\Bbb{Set}\ $ (just take the restriction of the hom functor). In the paper I show that bridges are just the so called Morita contexts connecting two profunctors.
A pair of adjoint functors $F\dashv G$ determines one single profunctor (up to isomorphism), namely $F_*:=\hom(F-,-)$ which is the same as $G_*:=\hom(-,G-)$.
This is the case for a profunctor (as category) iff $\Bbb A$ is coreflective and $\Bbb B$ is reflective subcategory therein.
The two profunctors corresponding to the two directions in your example arise from the forget forgetful functor $\Bbb{TopGr}\to\Bbb{Top}$. 
