Exponentials are the "closed" part of cartesian closed categories. The term "closed category" was first introduced in Eilenberg and Kelly's 1966 paper Closed Categories, where they're seen as generally the sort of thing that a category can be enriched over.
In the usual theory of categories, with any two objects $A$, $B$ of a category $\mathcal A$ there is associated a set $\mathcal A(A, B)$ of morphisms of $A$ into $B$. Frequently the set $\mathcal A(A, B)$ is endowed with an additional structure such as a privileged element or an abelian group structure.
[...]
In order to gain sufficient generality one should assume that $\mathcal A(A, B)$ is an object of some category $\mathcal V_0$, that this category $\mathcal V_0$ is equipped with a functor $V: \mathcal V_0 \to \mathcal S$ into the category $\mathcal S$ of sets, and that $V\mathcal A(A, B)$ is the set of morphisms $A \to B$ in $\mathcal A$. One then can write $\mathcal A_0(A, B)$ for $V \mathcal A(A, B)$, and distinguish the "enriched category" $\mathcal A$ from the ordinary category do that underlies it. Upon inspection it turned out that the categories $\mathcal V_0$ which occur in this connexion are endowed with a structure considerably richer than that of a category. We propose calling these "closed categories", and we may best describe them by citing two examples.
[...]
The basic elements of the structure of a closed category now become clear. First there is an ordinary category $\mathcal V_0$ [...]. Next there is a functor $V: \mathcal V_0 \to \mathcal S$. Then an internal Hom-functor $\mathcal V_0^{op} \times \mathcal V_0 \to \mathcal V_0$, denoted by $[A, B]$, and such that $V[A,B]$ is the set $\mathcal V_0(A, B)$ of morphisms $A \to B$. Further there is a unit object $I$ and a natural isomorphism $i: A \simeq [I, A]$.
(notation slightly altered for clarity)
The "closedness" of the category is that taking the morphisms between two objects gives another morphism in the same category, rather than the category of sets or some other category. This object of morphisms between two objects is the "internal Hom-functor" referred in the quote above and is precisely what the exponential is in cartesian closed categories.
For a complete definition of closed category, check out that paper (which uses fairly dated notation, but has lots of good examples) or the nLab page closed category.