What is the formal definition of indefinite integrals? I know the definition of definite integral using limits (or sups and infs) in rectangles in a given partition. However I realized that I don't know the formal definition of indefinite integrals.
My question is really simple, what is the definition of indefinite integrals?
 A: You can formally define the the concept of an anti-derivative and introduce the notation of an indefinite integral to mean the set of all anti-derivatives of a function. For the anti-derivative it would be something along the lines of:

For a function $f : [a,b] \to \mathbb{R}$, we call a differentiable function $F : [a,b] \to \mathbb{R}$ an anti-derivative of $f$ if $F'(x) = f(x)$ for all $x$ in $(a,b)$.

Then introduce the following notation for the set of all anti-derivatives of $f$:
$$\int f(x) \, \mbox{d}x$$
Or, once you have the definite integral, you can use it to introduce the indefinite integral, for example:
$$\int f(x) \, \mbox{d}x = \int_a^x f(t) \, \mbox{d}t + C $$
This relies on the fact that the definite integral in the RHS is one anti-derivative of $f$ (this is the fundamental theorem of calculus) and you use the fact that all anti-derivatives can only differ up to a constant term, so adding the arbitrary constant gives you all of them. 
Apostol does this in his Calculus, Vol. 1 where he introduces the definite integral long before the "Leibniz notation for the indefinite integral".
A: I think the concept you'll find more about in the literature is "anti-derivative" rather than "indefinite integral." In all the settings I can think of, the definition of an anti-derivative $F$ first requires a derivative, say $D$, to be defined, and then is defined as "$F$ is an anti-derivative of $f$ iff $DF = f$". The reason that you'll find more about "anti-derivative" rather than "indefinite-integral" is that "anti-derivative" generalizes, while indefinite-integral comes from the classical case of the fundamental theorem of calculus that others have mentioned:

First, define the derivative $\frac{d}{dt}$ of differentiable functions $f : [a,b] \to \mathbb R$. Then, an anti-derivative of $f$ is any $F : [a,b] \to \mathbb R$ such that $\frac{d}{dt} F = f$. The fundamental theorem of calculus tells us that some $F$ exists for all Riemann-integrable $f$, and for any anti-derivative $F$, we have, for $\alpha \leq \beta$ in $[a,b]$,
$$
\int_\alpha^\beta f(x) \, dx = \left. F(x) \right|_\alpha^\beta = F(\beta) - F(\alpha).
$$
The idea of "indefinite integral" stems from the above by "leaving off $\alpha$ and $\beta$", which is what's meant in writing
$$
\int f(x) \, dx = F(x).
$$
But the more precise way of writing the above is via anti-derivatives: $\frac{d}{dt} F = f$. 

Some generalizations of "anti-derivative" (but not necessarily "indefinite-integral") are found in functional analysis:


*

*A first concrete example: we say that $u \in L^2(\mathbb R)$ has a weak derivative ($u'$) in $L^2(\mathbb R)$ iff there exists $u' \in L^2(\mathbb R)$ such that the following integral equation holds:
$$
\left\langle u, - \frac{df}{dx} \right\rangle_{L^2} = \left\langle u', f \right\rangle_{L^2} \qquad\forall f \in C_c^1(\mathbb R).
$$
(You can make sense of this by integrating by parts.) In this case, we could instead say that $u' \in L^2(\mathbb R)$ has an anti-derivative ($u$) in $L^2(\mathbb R)$ if there exists $u \in L^2(\mathbb R)$ such that the same equation holds.

*More generally, "primitives" are solutions $u$ to the differential equation $\partial u = v$, where $u$ and $v$ are distributions. ($v$ is known, and $u$ is to be found.) We could just as well have called $u$ an anti-derivative of $v$ rather than a primitive. In this case, there may not always be a good/reasonable way to write this as an integral equation like $v = \int u$.

*Measures $\nu$ with a Radon-Nikodym derivative $f = \frac{d\nu}{d\mu}$ could just as well be called anti-derivatives of their Radon-Nikodym derivatives. In this case, writing something like $\nu = \int f \, d\mu = \int \frac{d\nu}{d\mu} \,d\mu$ makes sense, too.
