When is examining only one direction of a bijective map $\phi:X\to Y$ enough to categorize it as an isomorphism? Let $X,Y$ be sets with $\phi:X\to Y$ being bijective. If we consider $X$ and $Y$ various structures and ask what conditions do we have to impose on $\phi$ for it to be an isomorphism, we can break these cases up into two groups:
Group 1:


*

*If $X,Y$ are vector spaces, then $\phi$ needs to be linear. 

*If $X,Y$ are groups, then $\phi$ needs to respect the group operation.

*If $X,Y$ are metric spaces, then $\phi$ needs to be an isometry.

*etc.


Group 2:


*

*If $X,Y$ are topological spaces, then $\phi$ needs to be continuous and open.

*If $X,Y$ are smooth manifolds, then $\phi$ needs to be a diffeomorphism.

*etc.


In case one, we only need to verify that $\phi$ satisfies certain properties (that $\phi$ is linear, respects group operation, isometry, etc.). Whereas in group 2, we need to verify that both $\phi$ AND $\phi^{-1}$ satisfy certain properties (continuous, smooth, etc). My question is, what is it that distinguishes the members of group 1 verses group 2? Why is it enough to check a certain condition in one direction only enough in certain cases?  
My topology professor had the idea that for the types of structures in group 1, we examine these with first order logic (that is we are always discussing how $\phi$ interacts with particular elements of the sets), while the structures in group 2 are examined with second order logic. The way I understand this is that say for a topology, we only consider the open sets of $X$ and of $Y$ and not the actual elements of $X$ and $Y$ in order to establish if $\phi$ is an homeomorphism. While compared to the case of vector spaces, we have to consider the actual elements of $X$ and $Y$ to ensure that $\phi(ax_1+bx_2)=a\phi(x_1)+b\phi(x_2)$. But I have a problem with this theory: If we take the case of $X,Y$ being topological spaces, we can consider the topologies as sets and the open sets as being the members of the sets. Wouldn't this turn this into more of a "first order logic" scenario? 
Please note that I have (almost) no knowledge of category theory so I would really appreciate an answer that doesn't go too in-depth into that... unless of course that is the only appropriate way of answering this question. I also am certainly no logician, so I may be using some terms not entirely correctly but I would be more than happy to correct them if that is the case. 
 A: In the first class, homomorphisms must respect operations defined on all inputs from $X$:


*

*For vector spaces: addition: $u+v=w\implies \phi(u)+\phi(v)=\phi(w)$ and $\alpha u=v\implies \alpha\phi(u)=\phi(v)$

*For groups: $ab=c\implies\phi(a)\phi(b)=\phi(c)$

*For metric spaces: $d(x,y)=r\implies d(\phi(x),\phi(y))=r$
More generally, $F(x_1,\ldots, x_m)=x\text{ or }=c\implies F(\phi(x_1),\ldots, \phi(x_m))=\phi(x)\text{ or }=c$. Then bijectivity of $\phi$ lets us conclude that the same property holds for the inverse map $\psi$: E.g., if $F(y_1,\ldots, y_n)=y$, then $F(\psi(y_1),\ldots, \psi(y_n))$ is some $x$ and it must be the case that $\phi(x)=y$, hence $x=\psi(y)$ as desired.
This does not work for e.g. topological spaces, even if we define an infinitary operation $F(x_1,\ldots)=1$ or $=0$ if $\{x_1,\ldots\}$ is open or not. For respecting this $F$ would not define morphisms of $\mathsf{Top}$ (i.e., continuous maps). Recall that these are rather defined in a contra-variant fashion, i.e., the preimages of open sets in $Y$ have to be open in $X$.
A: In arguably all interesting cases you only need to define your type-1 operation and can recover your type-2 operation from that. Let's say we have such a type-1 operation, say, $$\phi: X \to Y,$$ where $X$, $Y$ are two arbitrary structures of the same kind. For example, $X$, $Y$ could be groups. Or they could both be vectorspaces. So we assume $\phi$ being such a compatible operation, e.g. for groups it's a group homomorphism, for vectorspaces it's a linear function.
Now I claim I can define your type-2 operation by that:

Definition (Isomorphism): We call a type-1 operation $\phi: X \to Y$ an isomorphism iff. there is a type-1 operation $\psi: Y \to X$ such that
  
  
*
  
*$\psi\circ\phi=id_X$ and
  
*$\phi\circ\psi=id_Y$.
  

Let's see if my notion of isomorphism coincides with your type-2 operations:


*

*
If $X,Y$ are topological spaces, then $\phi$ needs to be continuous, bijective and open.

Note that you forgot "bijective" in your post. If $\phi: X \to Y$ is continuous, it's a type-1 operation for topological spaces. Since it's bijective we have $\psi: Y \to X$ as the inverse function. Does that $\psi$ fit my isomorphism definition? First, is it a type-1 operation at all? Yes, it is! Openness of $\phi$ implies continuity of $\psi$. Next: do we have those two equalities with $id_X$ and $id_Y$? Yes, since they are inverse functions by construction.
So a type-2 operation for topological spaces is exactly a type-1 operation which happens to be an isomorphism as defined above.

*
If $X,Y$ are vectorspaces, then $\phi$ needs to be linear and bijective.

Left as an exercise for the reader.
You see, there's no need to specify what a type-2 operation is for top. spaces or vectorspaces. It's induced by the definition of the type-1 operation. And that's the case for almost all interesting structures! 
To answer your question:

When is examining only one direction of a bijective map $\phi: X \to Y$ enough to categorize it as an isomorphism?

The classification as an isomorphism almost always exhibits an inverse $\psi$ as stated above. It's just the case that in some structures, we have theorems relating a property of $\phi$ to the existence of $\psi$:


*

*for ordinary functions solely on sets, bijectivity ensures existence of inverse $\psi$

*for groups, bijectivity ensures existence of inverse $\psi$

*for topological spaces, bijectivity + openness ensure existence of inverse $\psi$
(In all cases I assume that $\phi$ is already a type-1 operation.)

Categorical Takeaway
You see that independent of the mathematical field (group theory or topology), the notion of isomorphism corresponds to the very general definition I gave above. That's one driving point of category theory. There we define a category by specifying


*

*which objects it has (e.g. it contains all groups)

*and the type-1 operations (e.g. groups are connected via group homomorphisms).


The type-1 operation is called morphism. The notion of isomorphism is then as follows for every category:

Definition (Isomorphism in Categories): In a given category a morphism $\phi: X \to Y$ is called an isomorphism iff. there is a morphism $\psi: Y \to X$ in the same category such that
  
  
*
  
*$\psi\circ\phi=id_X$ and
  
*$\phi\circ\psi=id_Y$.
  

