Prove that $(a,b)$ is homeomorphic to $(0,1)$ 
Proposition:
If $a$ and $b$ are real numbers with $a< b$, then $\left ( a,b \right )$ is homeomorphic to $\left ( 0,1 \right )$.

Proof:
Define
$$f:\left ( a,b \right )\rightarrow \left ( 0,1 \right) ,\ \ \ x \mapsto f\left ( x \right )=\frac{x-a}{b-a}$$
The inverse is $f^{-1}\left ( x \right )=\left ( b-a \right )x+a$ so a bijection exists.
It now suffices to show that $f$ and $f^{-1}$ are both continuous.
Recall:
The collection $\textbf{B}$ of intervals $\left ( a,b \right ) \subseteq \mathbb{R}$ is a basis for the standard topology on $\mathbb{R}$.
$\left ( 0,1 \right )\subseteq \mathbb{R}$ and so $\left ( s,t \right )\subseteq \mathbb{R}$.
Thus, the collection of intervals $\left ( s,t \right ) \forall s,t \in \left ( 0,1 \right )$ is a basis for the topology on $\left ( 0,1 \right )$.
In other words, we have a topology generated by a basis $\textbf{B}$.
It is true that $\forall s,t \in \left ( 0,1 \right ): \left ( s,t \right )$ is an open set.

Recall: a function $f: \left ( X,\tau \right )\rightarrow \left ( Y,U \right )$ is continuous IFF $\forall$ u in $U: f^{-1}\left ( u \right ) \in \tau$.

$f^{-1}\left ( s,t \right )=\left ( s\left ( b-a \right )+a,t\left ( b-a \right )+a \right )$.
How should I take it from here?
 A: The definition of continuity that you are referring to, at the second part of your post, is the general definition of continuity for maps betwenn arbitrary topological spaces. 
However, I would notice that you really do not need this: you have already shown that $f$, $f^{-1}$ are linear functions of the variable $x$ and linear functions are in fact continuous functions (by elementary arguments). So you are done.  
P.S.: If you feel that you have to resort to the general definition, I think it is enough to notice that since $t>s\Rightarrow t\left ( b-a \right )+a>s\left ( b-a \right )+a \ $, then the inverse image of an open interval $(t,s)$, that is:
$$
f^{-1}\left ( s,t \right )=\left ( s\left ( b-a \right )+a,t\left ( b-a \right )+a \right )
$$
is an open interval. Now since, as you have already mentioned in your post, the open intervals constitute bases for the topologies (thus any open set can be written as a union of open intervals) you can easily conclude that the inverse image of any open set is an open set. So you are done (again). 
