$\mathscr T_X$ will denote the set of all functions from a non-empty set $X$ into itself, with the binary operation of composition $\circ$ making it a semigroup, called the full transformation semigroup on $X$.
Is there a topology on the set $\mathscr T_X$ such that $\circ:\mathscr T_X\times \mathscr T_X\longrightarrow \mathscr T_X$ is a continuous function with respect to the product topology on $\mathscr T_X\times \mathscr T_X?$
(i.e., is there a topology on $\mathscr T_X$ making it a topological semigroup?)
Clearly, two (one if $X$ is a one-element set) topologies always work: the discrete and the indiscrete topology on $\mathscr T_X$ make the composition continuous as any function into an indiscrete space is continuous and any function from a discrete space is. (And the product of two discrete spaces is discrete.) I will call those two topologies trivial.
These topologies don't seem useful at all, so I will re-write the question. Let $\operatorname{card}(X)>1.$
Is there a non-trivial topology on $\mathscr T_X$ making it a topological semigroup?
(or at least, is there a construction of such a topology depending on $\operatorname{card}(X)$ which yields non-trivial topologies at least in some cases?)
I cannot think of any general approach to this question and I think I may not have the tools -- I know virtually nothing about topological semigroups. I would be grateful for any help, be it in the form of a hint, a reference, or a full or partial answer to the question. Also, please don't hesitate to comment on anything even remotely related to this.
Re Tara B's answer
I may be mistaken but I think your example works only for finite sets. In general, I think, when we have a semigroup $S$ and a non-empty proper subset $A\subset S,$ then $\{\emptyset, A,S\}$ forms a good topology iff the following two conditions are satisfied:
$(1)$ $A$ is a subsemigroup of $S;$
$(2)$ $S\setminus A$ is an ideal in $S.$
Let's say that in this situation, we call $A$ saturated and $S\setminus A$ prime. (I'm not sure if this is standard nomenclature, but I can imagine it might be.)
Suppose a non-empty proper subset $A\subset \mathscr T_X$ is saturated. Then there is $\phi\cdot\operatorname{id}=\phi\in A$ and so $\operatorname{id}\in A.$ Let $\psi\in S_X.$ Then $\psi\psi^{-1}=\operatorname{id}\in A,$ and so $\psi\in A.$ Therefore $S_X\subseteq A.$
But also, let $\mathscr T_X\ni\chi\mathscr J\operatorname{id}.$ Then for some $\alpha,\beta\in\mathscr T_X,$ we have $\alpha\chi\beta=\operatorname{id}.$ Hence $\alpha\chi\in A,$ and so $\chi\in A.$ Therefore the $\mathscr J$-class $J_{\operatorname{id}}$ is contained in $A.$
But for an infinite set $X,$ we have the proper containment $S_X\subsetneq J_{\operatorname{id}},$ because there are functions from $X$ to $X$ whose rank is equal to $\operatorname{card}(X)$ but which aren't permutations. So $S_X$ cannot be saturated.
I think for an infinite set $X$ there will be no such $A$ at all. I'm unable to prove this but I think we can obtain any function in $\mathscr T_X$ by composing functions in $J_{\operatorname{id}}.$ If that's true, then if $A$ were saturated, then it would be a subsemigroup containing a set generating the whole $\mathscr T_X$ and so $A=\mathscr T_X.$
$S_X$ clearly works for finite sets $X$ though. It's a subsemigroup of $\mathscr T_X$ and its complement is an ideal because the composition of functions of which at least one doesn't have the maximal rank cannot have the maximal rank either. And for finite $X,$ a function $\phi: X\longrightarrow X$ is a permutation iff it has the maximal rank. I believe $S_X$ is the only saturated subsemigroup of $\mathscr T_X$ for $X$ finite.
EDIT The statement
"when we have a semigroup $S$ and a non-empty proper subset $A\subset S,$ then $\{\emptyset, A,S\}$ forms a good topology iff the following two conditions are satisfied: $(1)$ $A$ is a subsemigroup of $S;$ $(2)$ $S\setminus A$ is an ideal in $S.$"
is false. When a semigroup isn't a monoid it may not be true. For example, Let $\mathbb N=\{1,2,\ldots\}$ be the additive semigroup of natural numbers. Let $A=\{1\}.$ Then $\{\emptyset,A,\mathbb N\}$ is a good topology on $\mathbb N,$ because the inverse image of $A$ under addition is empty. This is impossible in a monoid. It is also impossible in a monoid for the inverse image of $A$ to be equal $S\times S$ because the image of $S\times S$ under the semigroup operation is equal to $S.$ I have to think about it some more.