Which branch of mathematics rigorously defines infinitesimals? I have some trouble doing standard computations in calculus because of the notion of a differential, otherwise known as an infinitesimal, being rather ill defined, in my experience.
Are there any fields of mathematics that someone can recommend for trying to come to a more rigorous grip on the notion of infinitesimals? I've heard and read up about non-standard analysis, but from what I can tell, even the rigour of non-standard analysis isn't as firm as that of more established branches of mathematics. How accurate is this perception?
Any help is appreciated, thank you.
 A: Amusingly, one of the answers to the question you ask is that elementary calculus rigorously defines infinitesimals.
How does it do so? Via the notion of differential. The problem you're struggling with is almost backwards; the standard, traditional track is:


*

*Define the notion of derivative

*Use multivariable derivatives to define the notions of (tangent) vector and differential

*Conceptualize a notion of an "infinitesimal" neighborhood of a point


To elaborate on that last point, you're supposed to envision the points of the infinitesimal neighborhood to be enumerated by tangent vectors — the intuitive idea is that you take an "infinitesimal" step proportional to the tangent vector. Differentials are the functions on the infinitesimal neighborhood.
But this conceptualization is not trying to define anything new — it is merely a way of thinking about calculus. (albeit a very useful one!)
But the second point is bog standard. For example, in multivariable calculus, one incarnation of these notions is


*

*Tangent vectors to points in $\mathbb{R}^n$ are $n \times 1$ column vectors — the sort of thing you get when differentiating a vector function of one variable

*Differentials at points in $\mathbb{R}^n$ are $1 \times n$ row vectors — the sort of thing you get when differentiating a scalar function of $n$ variables


Both the subjects of differential geometry and algebraic geometry treat this sort of thing much more explicitly and more in-depth.
A: Your perception is wrong. Non-standard analysis is grounded on Logic and it's as solid as any other field of Mathematics. I suggest that you read Abraham Robinson's Non-standard Analysis.
A: Most objections to non-standard analysis seem to be about the use of the axiom of choice in the construction of the field of hyperreals. Non-standard analysis is completely rigorous, but if you're a hardcore constructivist then you may be a bit squeamish about it. Then again, there's always some things you need to take on faith in any branch of maths:


*

*If you're a hardcore finitist then you have to be really careful about analysis in general, since the conventional $\mathbb{R}$ as an object doesn't exist at all.

*If you don't accept the axiom of dependent choices then you're pretty limited in what you can do in real analysis, because many arguments rely on taking a sequence chosen arbitrarily.

*If you don't believe there is a nonprincipal ultrafilter on $\mathbb{N}$ then you can't construct the ultrapower required to create the hyperreals.


If you choose to allow more axioms ("there is an infinite set", "dependent choices", "there is a nonprincipal ultrafilter on $\mathbb{N}$") then you get access to correspondingly more interesting things you can do, but it's all still rigorous.
Note, however, that if you accept Choice then in a certain sense "anything you can do in non-standard analysis, you can also do without the hyperreals" (see https://math.stackexchange.com/a/51480/259262). It's an extra proof technique to make things easier by hiding many of the $\forall \exists$ quantifiers, rather than allowing you to prove genuinely new things that you couldn't prove before.
A: To answer the title question: "Which branch of mathematics rigorously defines infinitesimals?": the answer is "a serious undergraduate course in algebra, including the existence of a maximal ideal".  Indeed, that is all that's required to construct a hyperreal extension of the real field satisfying the requisite properties, such as existence of infinitesimals, etc.
A: Like others have pointed out, nonstandard analysis is perfectly rigorous, just a limited more complicated to set up. However, I will list some other, equally rigorous approaches to infinitesimals.


*

*Dual numbers: These are very simple to set up. A dual number is basically just a ordered pair of numbers. It is based on an infinitesimal called $\epsilon$, which is a number such that $\epsilon^2 = 0$. Their relation to differentiation is $f(a + b\epsilon) = f(a) + bf'(a)\epsilon$. In fact, if you use a programming language that implements dual numbers (like Haskell), you can find the derivative of any function by solving that instead of approximating the derivative with a secant line. However, do not expect to get very far in Calculus with them: they only help you define derivatives, they have no transfer principle, and they are not even a field.

*Surreal numbers: This fun little number system includes real numbers, infinitesimals, cardinal number, ordinal numbers, and every ordered field. It is also satisfies the field axioms itself. However, they have nearly no applications in Calculus, because they are too different from real numbers. They were actually invented for use in game theory, their seminal text being On Numbers and Games (which is a great book, by the way). They also have an extremely simple definition given their complexity, even simpler than the real numbers. Conway actually proposes that it would be simpler to construct the real numbers by first constructing the surreal numbers, and then getting rid of all the non-real surreal numbers.

A: There's an approach, by the names of synthetic differential geometry or smooth infinitesimal analysis, which considers a kind of 'real line extended with' nilpotent elements, in this case, objects $ d $ not necessarily equal to $ 0 $ satisfying
$ d^2 = 0 $
Note this implies one's no longer working with/inside a conventional algebraic field, and indeed the axiomatic development of this theory, though in some sense simpler than the more usual forms of Analysis, demands some specially careful attention 
