If $b$ is the largest member of $(X,\leq)$. how does $(a,b]$ become basic open set? Let $(X,\leq)$ be a well-ordered set, Let $\mathscr T$ be the order topology on $X$.
How do I prove that every interval of the form $(a,b]$ is open in $X$?
Proof is given in the textbook:
Case 1:
If $(X,\leq)$ has the largest member and $b$ is that member, then $(a,b]$ is the basic open set. 
Case 2:
If $b$ is not a largest member of $(X,\leq),$ Let $A=\{x\in X:x>a\}$. Since $(X,\leq)$ be a well-ordered set, $A$ has a least element $c$. Thus $(a,b]=(a,c)$ and Hence $(a,b]$ is open.
Definition of order topology on $X$ given in the text as follows.
Definition Let  $(X,\leq)$ be a linearly ordered set, and let $\mathscr S$ be the collection of all subsets of the form $\{x\in X:x<a\}$ and $\{x\in X:x>a\}$, where $a\in X$. Then the topology that has $\mathscr S$  subbasis is called the order topology on $X$.
Doubts:


*

*If $b$ is the largest member of $(X,\leq)$, how does $(a,b]$ become basic open set?

*Case 2: $(a,b]$ is open. since $(a,b]=(a,c)=\{x\in X:x>a\}\cap \{x\in X:x<c\}.$ $\{x\in X:x>a\}\in \mathscr S\subseteq \mathscr T$ and $\{x\in X:x<c\}\in \mathscr S\subseteq \mathscr T\implies \{x\in X:x>a\}\cap \{x\in X:x<c\}\in \mathscr T $. Am I correct?
 A: *

*$(a,b]=\{x\in X\,|\,x>a\}\in \mathscr S\subset \mathscr T$

*Yes, you are correct.

A: There are a few different definitions of "subbasis," and without knowing exactly which version your text uses, the approach may vary.
The most common I've seen are as follows:

  
*
  
*Given a set $X,$ a topology $\mathcal T$ on $X,$ and a collection $\mathcal S$ of subsets of $X,$ we say that $\mathcal S$ is a subbasis for $\mathcal T$ if $\mathcal T$ is the set of unions of finite intersections of elements of $\mathcal S.$ (Here, the empty intersection $\bigcap\emptyset$ is taken to be $X.$)
  
*Given a set $X,$ a topology $\mathcal T$ on $X,$ and a collection $\mathcal S$ of subsets of $X,$ we say that $\mathcal S$ is a subbasis for $\mathcal T$ if the collection of finite intersections of elements of $\mathcal S$ comprise a basis for $\mathcal T.$
  
*Given a set $X,$ a topology $\mathcal T$ on $X,$ and a collection $\mathcal S$ of subsets of $X,$ we say that $\mathcal S$ is a subbasis for $\mathcal T$ if $\mathcal T$ is the coarsest topology on $X$ that contains $\mathcal S$--that is, if $\mathcal S\subseteq\mathcal T,$ and given any topology $\mathcal T'$ on $X$ with $\mathcal S\subseteq\mathcal T'$ we have $\mathcal T\subseteq\mathcal T'.$

The three definitions above are equivalent. Another common one is:


  
*Given a set $X,$ a topology $\mathcal T$ on $X,$ and a collection $\mathcal S$ of subsets of $X,$ we say that $\mathcal S$ is a subbasis for $\mathcal T$ if $X=\bigcup\mathcal S$ and $\mathcal T$ is the coarsest topology on $X$ that contains $\mathcal S.$

If $X$ has at least two points and we are considering topologies in which one-point subsets of $X$ are closed, then this definition is equivalent to the others, but need not be, in general. Fortunately, in an order topology, one-point subsets will be closed, so the definition is largely immaterial. Under the first two definitions, it is readily shown that any element of $\mathcal S$ will be an element of $\mathcal T,$ and in the other two definitions this is already explicit.
In particular, if $b$ is the largest element of $X,$ then we'll have $(a,b]=\{x\in X:x>a\},$ which is then an element of the given subbasis, so an open set.

I note that there is an error in the Case 2 proof you site from your book. It should instead consider the set $A=\{x\in X:x>b\}.$ Then $A$ is non-empty by the Case 2 assumption, and so has the least element $c$ used in the rest of that proof.
As for justifying why $(a,c)\in\mathscr T,$ that works perfectly! It follows a bit more immediately from the first and second definitions of subbasis mentioned above, but only slightly.
