How did early mathematicians make it without Set theory? It is said that Cauchy was a pioneer of rigour in calculus and a founder of complex analysis. Yet if baffles me as set theory was an invention of the 1870s, 20 years after the death of Cauchy. Currently the beginning of most concepts in mathematics begins with the concept of set. Furthermore the concept of groups whose foundations were laid by Galois and Abel were done so long before set theory.
I hope there is a genral way to answer these questions
1) We define functions with a domain and range both being sets. But when Cauchy used the symbol 'f(x)', what did it really mean to him? As Cauchy was notorious for his rigorous approach, it is hard to believe that he may have just used the word function ambiguously with intuitive satisfaction.
(If the following question makes the topic too broad I'd be more than happy to list it as a separate question.
2)To a certain extent I can even brush away the idea of functions before sets. But I simply cannot grasp how the concept of group was formulated without a set and I'm puzzled as to how Galois and Abel were independently able to frame methods to prove the unsolvability of the quintic (these days the proof makes generous use of set theory)without sets.
In these days where N, Z, Q and R all sets, how did the early masters do what they did? How on earth was calculus made rigorous without the sets of different numbers?
 A: The term "set theory" as used in the phrase "set theory was invented in the late nineteenth century" doesn't just refer to "the concept of a collection of objects". Of course people have always understood the notion of taking a collection of things and referring to them by a single name. What was new in the nineteenth century was the creation of a formal framework for talking about much more complex problems involving infinite sets - cardinality, for example, or theorems like Zorn's lemma. The Continuum Hypothesis. I don't need ZFC to talk about the set of fingers on my left hand, to conclude that that set contains five elements, or that its union with the set of fingers on my right hand contains ten elements. So this is a bit like asking "How did ancient man know that objects were solid without knowing about electrostatic force?".
Mathematicians in Cauchy's time thought of functions as "rules" that assigned one output number to each input number. I'm not sure why you think modern set theory is needed in order to be able to talk about such a concept.
As for Galois and Abel, the notion of a group as a set with axioms imposed on it didn't exist until much later, although I'm sure they wouldn't have had any issue with such a definition, other than that they might not have seen any motivation for it. They thought of groups as permutation groups on a (finite) set of solutions to an equation - because again, not having a precise notion of the Axiom Of Extensionality or the undecidability of CH doesn't prevent anyone from talking about swapping around elements in a finite set of objects.
A: An illuminating example is provided by algebraic structures. Let's examine Cauchy's explanation of his construction of the ring of complex numbers $\,\Bbb C$ (excerpted from my answer here).
A major accomplishment of the set-theoretical definition of 
algebraic structures was to eliminate imprecise syntax and semantics. 
Eliminating the syntactic polynomial term $\rm\ a+b\cdot x+c\cdot x^2\  $ 
and replacing it by its rigorous set-theoretic semantic reduction 
$\rm\:(a,b,c,0,0,\ldots)\:$ eliminates many ambiguities. There can no longer be any doubt about the precise denotation of the symbols $\rm\: x,\; +,\;\cdot\:,$ or about the meaning of equality of polynomials, since, by set theoretic definition, tuples are equal iff their components are equal. The set-theoretic representation ("implementation") of these algebraic objects gives them rigorous meaning, reducing their semantics to that of set-theory.
Similarly for complex numbers  $\rm\,a + b\cdot {\it i}\ $ 
and their set-theoretic representation by Hamilton as pairs of reals $\rm\,(a,b).\,$ Before Hamilton gave this semantic reduction of $\,\mathbb C\,$ to $\Bbb R^2,\,$ prior syntactic constructions (e.g. by Cauchy) as 
formal expressions or terms $\rm\:a+b\cdot {\it i}\:$ were subject to heavy criticism regarding 
the precise denotation of their constituent symbols, e.g. 
precisely what is the meaning of the symbols $\rm\;{\it i},\, +,\, =\,?\, $ 
Said in more modern language, Cauchy essentially constructed $\mathbb C$ as $\,\Bbb R[x]\,$ mod $\,x^2+1,\,$ which nowadays we reify structurally as the
the quotient ring $\rm\:\Bbb C\cong \mathbb R[x]/(x^2+1)\cong \Bbb R[{\it i}].\,$ Cauchy attempted to present this in the language of congruences (polynomial modular arithmetic). However, in Cauchy's time 
mathematics lacked the necessary (set-theory) foundations to 
rigorously define the syntactic expressions comprising the
polynomial ring term-algebra $\rm\mathbb R[x]$, and its quotient ring of
congruence classes $\rm\:(mod\ x^2+1).\,$ The best that Cauchy could 
do was to attempt to describe the constructions in terms of 
imprecise natural (human) language, e.g, in $1821$ Cauchy wrote: 

In analysis, we call a symbolic expression any combination of 
  symbols or algebraic signs which means nothing by itself but 
  which one attributes a value different from the one it should 
  naturally be [...] Similarly, we call symbolic equations those 
  that, taken literally and interpreted according to conventions 
  generally established, are inaccurate or have no meaning, but 
  from which can be deduced accurate results, by changing and 
  altering, according to fixed rules, the equations or symbols 
  within [...] Among the symbolic expressions and equations 
  whose theory is of considerable importance in analysis, one 
  distinguishes especially those that have been called imaginary. $\quad$ -- Cauchy, Cours d'analyse,1821, S.7.1

While nowadays, using set theory, we can rigorously interpret such "symbolic expressions" 
as terms of formal languages or term algebras, it was far too 
imprecise in Cauchy's time to have any hope of making sense 
to his colleagues, e.g. Hankel replied scathingly: 

If one were to give a critique of this reasoning, we can not 
  actually see where to start. There must be something "which 
  means nothing," or "which is assigned a different value than 
  it should naturally be" something that has "no sense" or is 
  "incorrect", coupled with another similar kind, producing 
  something real. There must be "algebraic signs" - are these 
  signs for quantities or what? as a sign must designate something 
  - combined with each other in a way that has "a meaning." I do 
  not think I'm exaggerating in calling this an unintelligible 
  play on words, ill-becoming of mathematics, which is proud 
  and rightly proud of the clarity and evidence of its concepts. $\quad$-- Hankel

Thus it comes as no surprise that Hamilton's elimination 
of such "meaningless" symbols - in favor of pairs of reals - 
served as a major step forward in placing complex numbers on a 
foundation more amenable to his contemporaries. 
Although there was not yet any theory of sets in which to 
rigorously axiomatize the notion of pairs, they were far easier 
to accept naively - esp. given the already known closely 
associated geometric interpretation of complex numbers. 
Hamilton introduced pairs as 'couples' in $1837$ [1]: 

p. 6: The author acknowledges with pleasure that he agrees with 
  M. Cauchy, in considering every (so-called) Imaginary Equation 
  as a symbolic representation of two separate Real Equations: 
  but he differs from that excellent mathematician in his method 
  generally, and especially in not introducing the sign  sqrt(-1) 
  until he has provided for it, by his  Theory of Couples, 
  a possible and real meaning, as a symbol of the couple (0,1) 
p. 111:  But because Mr. Graves employed, in his reasoning, the 
  usual principles respecting about Imaginary Quantities, and 
  was content to prove the symbolical necessity without showing 
  the interpretation, or inner meaning, of his formulae, the 
  present Theory of Couples is published to make manifest that 
  hidden meaning: and to show, by this remarkable instance, that 
  expressions which seem according to common views to be merely 
  symbolical, and quite incapable of being interpreted, may pass 
  into the world of thoughts, and acquire reality and significance, 
  if Algebra be viewed as not a mere Art or Language, but as the 
  Science of Pure Time.  $\quad$ -- Hamilton, 1837

Not until the much later development of set-theory was it explicitly realized 
that ordered pairs and, more generally, n-tuples, serve a fundamental foundational role, providing the raw materials necessary to construct composite (sum/product) structures - the raw materials required for the above constructions of polynomial rings and their quotients.
Indeed, as Akihiro Kanamori wrote on p. 289 (17) of 
his very interesting paper [2] on the history of set theory: 

In 1897 Peano explicitly formulated the ordered pair using 
  $\rm\:(x, y)\:$ and moreover raised the two main points about the 
  ordered pair: First, equation 18 of his Definitions stated 
  the instrumental property which is all that is required of 
  the ordered pair: 
$$\rm (x,y) = (a,b) \ \ \iff \ \ x = a \ \ and\ \ y = b $$
Second, he broached the possibility of reducibility, writing: 
  "The idea of a pair is fundamental, i.e., we do not know how 
  to express it using the preceding symbols." 

Once set-theory was fully developed one had the raw materials 
(syntax and semantics) to provide rigorous constructions of 
algebraic structures and precise languages for term algebras. The polynomial ring $\rm\:R[x]\:$ is nowadays just a special case of much more general constructions of free algebras. Such equationally axiomatized algebras and their genesis via so-called 'universal mapping properties'  are topics discussed at length in any course on Universal Algebra - 
e.g. see Bergman [3] for a particularly lucid presentation. 
[1] William Rowan Hamilton. Theory of conjugate functions, or algebraic couples; with a preliminary and elementary essay on algebra as the science of pure time
Trans. Royal Irish Academy, v.17, part 1 (1837), pp. 293-422.)
http://www.maths.tcd.ie/pub/HistMath/People/Hamilton/PureTime/PureTime.pdf
[2] Akihiro Kanamori. The Empty Set, the Singleton, and the Ordered Pair
The Bulletin of Symbolic Logic, Vol. 9, No. 3. (Sep., 2003), pp. 273-298.
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.95.9839
PS  http://www.math.ucla.edu/~asl/bsl/0903/0903-001.ps
PDF http://ifile.it/b20c48j 
[3] George M. Bergman. An Invitation to General Algebra and Universal Constructions.
PS  http://math.berkeley.edu/~gbergman/245/
PDF http://ifile.it/yquj5w1 
A: Some hints...
You can start reading Cauchy's elucidation of function (1823) :

On nomme quantité variable celle que l'on considère comme devant recevoir successivement plusieurs valeurs différentes les unes des autres. On appelle au contraire quantité constante toute quantité qui reçoit une valeur fixe et déterminée.

[We name variable a quantity that receives successively many different values. We name constant a quantity that receives a fixed and determined value.]

Lorsque des quantités variables sont tellement liées entre elles, que, la valeur de l'une d'elles étant donnée, on puisse en conclure les valeurs de toutes les autres, on conçoit d'ordinaire ces diverses quantités exprimées au moyen de l'une d'entre elles, qui prend alors le nom de variable indépendante; et les autres quantités, exprimées au moyen de la variable indépendante, sont ce qu'on appelle des fonctions de cette variable .

[When some variable quantities are linked together in a way that, having fixed the value of one of them, all others quantities can be determined, on conceive these different quantities as expressed by way of one of them, named independent variable. The remaining quantities, expessed by way of the independent variable, are named functions of that variable.]


Thus, in a nutshell, the concept of "function" was a primitive one, like today for set. A function is a correspondence (a relation) between two "variable quantities".

It is worth noting that Cauchy's definition of "variable quantity" was already present into de L'Hôpital's textbook : Analyse des infiniment petits pour l'intelligence des lignes courbes (1st ed, 1696), the first calculus' textbook. See :

*

*Robert Bradley & Salvatore Petrilli & CEdward Sandifer (editors), L’Hôpital's Analyse des infiniments petits (2015).

An early occurrence of "function" is in Leibniz, in De linea ex lineis numero infinitis ordinatim ductis inter se concurrentibus formata, easque omnes tangente, ac de novo in ea re Analysis infinitorum usu (1692), but a "reasonable" definition of function is available only with Johann Bernoulli, Remarques sur ce qu'on a donne jusqu'ici de solutions des problemes sur les isopdrimitres (1718) and Leonhard Euler, Introductio in analysin infinitorum (1748).

Regarding group we may see e.g. Arthur Cayley : he uses the name "set" in his definition of group (1854) :

A set of symbols : $1,α,β,\ldots$ all of them different, and such that the product of any two of them (no matter in what order), or the product of any one of them into itself, belongs to the set, is said to be a group.

Set here is not a mathematical object : no specific properties of sets are assumed.
A: Set theory is one of the most common ways in modern mathematics to justify the ontology of the entities mathematicians are dealing with, by providing a general framework where all (or most, to be safe) of these entities can be constructed from the empty set. On the other hand, the procedures of mathematics are able to stand, and rigorously at that, without the set-theoretic foundations. Cauchy was rigorous not in the sense that he had set theory (he didn't) or that he gave epsilon-delta definitions of continuity (he didn't), but because he adhered to the rigor of the geometry of Euclid as it was practiced for several centuries before him, and because he rejected the generality of algebra (roughly, cavalier summation of divergent series) of Euler and Lagrange. The distinction procedures versus ontology and how it sheds light on the history of mathematics is explored in the article "Toward a history of mathematics focused on procedures".
A: I am certainly not an expert on this topic but it is not that the mathematicians did not know about the concept of set before the beginnings of Set Theory. Surely they did know about the concept of set and about various kinds of sets that appear in mathematics, it is only that, as far as I know, before the beginnings of Set Theory that there were no formalized and organized theories that deal only with sets and with functions defined on them and with operations that can be defined on some "sets of sets".
If we take, for example, the concept of group, you can see that it is defined as: "the set G, together with the operation * such that operation * satisfies this and this and this and this axiom", so in the process of axiomatization there does not exist the need to tell a lot of set-theoretical stuff about the set on which the operation(s) are defined (at least not in those mathematical structures that are familiar to me).
You should be able to see that to resolve some mathematical problems you almost do not need Set Theory at all, so, at least for me, the importance of Set Theory is not so much in its ability to add more rigour to some mathematical definitions but in that that it gives us some results with which we can do some things we were not able to do before.
A: I recently read a scholarly book relevant to this question.

Jeremy Gray, Plato's Ghost: The Modernist Transformation of Mathematics

The transformation in question goes from 1880 to 1910 (roughly speaking).  Gray discusses how mathematics was done before, the turmoil surrounding the change, and how it was done after.  
Recommended.  But not for the faint of heart.
A: We can get a hint to an answer by considering some of today's undergraduate textbooks which do not assume a knowledge of set theory, and do not include a formal definition of a function. There are still a lot of such books. 
An entire calculus course can be taught without dwelling on set theory. In fact, I know that if I have to teach calculus to a big class, the worst thing I can do is to spend half an hour on set theoretical issues; I can see the glaze settle over their eyes (except for the small minority of students who instead get a glint of excitement in their eyes, and who I am constantly on the lookout for).
A student can ingest what they need to know about derivatives and their applications, integrals and their applications, differential equations, et cetera, and that student can go on to apply calculus in many different branches of inquiry, without me ever spending that half hour on set theoretical issues. And that's a good thing. 
