Are there other approaches for the foundations of mathematics, other than logic and set theory? Are there other approaches for the foundations of mathematics, other than logic and set theory?
And why does set theory begin talking about objects and groups of objects.
Is it proven somewhere that that is the most fundamental concept?
What is the main idea behind set theory? I do understand it, but I try
to get the bigger picture, in what way does it try to set up the
foundations of mathematics? It would be good to compare it to other approaches.
 A: On one hand, how much more basic and foundational can you get than objects and sets of objects?  Of course, I say that as someone who comes squarely from the logic and set theory camp.  On the other hand, you do ask a really good question, especially if you want to consider approaches to objects and sets other than the Zermelo-Frankel axioms.  For example, you may want to look at Russell's type theory or Quine's New Foundations.  Actually getting away from explicitly talking about objects and sets, though, the only thing I can think of is category theory, and in particular topoi.  That may seem like a bit of a cheat, though -- the idea you start with in topoi is to consider the elementhood relation for sets as an "arrow" in the category sense, so you don't really get away from sets as much as just model them differently.
A: I will try to answer just one of the questions:

in what way does [set theory] try to set up the foundations of mathematics?

It should help to see specific examples of how concepts from other areas of math can be formalized in set theory.  Let's consider two examples.
(1) A group can be considered as an ordered pair of objects $(G, f)$ where $G$ is a set and $f$ is a function $G \times G \to G$ satisfying certain logical properties.  The Cartesian product $G \times G$ is the set of ordered pairs of elements of $G$.  A function $G \times G \to G$ is a subset of the Cartesian product $(G \times G) \times G$ satisfying certain properties.  The notion of "ordered pair" can be formalized in terms of (unordered) sets using Kuratowski's definition.
(2) A real number can be defined as a set of rational numbers satisfying a certain logical property (that of being a Dedekind cut.)  A rational number in turn can be defined as a set of ordered pairs of integers, namely an equivalence class under a certain equivalence relation.  Similarly, integers can be defined as equivalence classes of ordered pairs of natural numbers.  Natural numbers can be defined as certain kinds of von Neumann ordinals, which are sets.
In both cases (1) and (2) the reductions to sets are not very elegant, but they get the job done. I think it is remarkable that we are able to do such a thing at all.  You could try to reduce set theory and group theory to analysis instead, or perhaps reduce analysis and set theory to group theory, but I don't think you would find as much success.
Of course I don't claim to have shown here that some other approach to the foundations of mathematics, such as category theory, would not work just as well.
A: The main idea of set theory, as I see it: At the very least, a set theory presents (1) a fixed number of ways that you can infer the existence of a set given the supposed existence of another set(s) (e.g. axioms for subsets, power sets, Cartesian products), and (2) definitions of equality for sets and ordered n-tuples. 
Using such axioms, along with the rules of logic, it is possible, in theory, to derive all of modern number theory and classical analysis starting with nothing more than the presumed existence of just one Dedekind-infinite set. 
