What a reference frame is mathematically? Physicists usually talk about reference frames and more specially inertial reference frames. This is particularly important in Mechanics and Relativity. Now, from the Physics standpoint there's no doubt on what is a reference frame: it is a point of view to observe a phenomenon. From the Math standpoint, however, usually books speak very loosely about reference frames.
There are many things that books imply without ever making it explict. Some of those things are:


*

*Sometimes reference frames seems to be considered simply as coordinate systems on spacetime

*But sometimes, books seems to stress that they are "sets of axes", so only cartesian coordinates on flat spacetime would be reference frames

*Usually reference frames seems to be able to move around, but coordinate systems (on Differential Geometry sense) can't do this


Now, these are just some of the concerns. Never it is made clear what really a reference frame is mathematically and this is annoying me for a long time. Some mechanics books do even worse: they make it seems that a reference frame is just a question of interpreting the equations correctly.
Just making the last point clearer: in Differential Geometry one coordinate system is suited for a particular subset, so it doen't make sense to try moving one coordinate system around. This could only work in $\mathbb{R}^n$.
Also, for any manifold $M$ we can consider the bundle of frames $F(M)$ which is the bundle whose fiber at $a \in M$ is homeomorphic to $GL(n,\mathbb{R})$ representing all the possible bases for the tangent space at $a$. Because of the name (bundle of frames) I thought this could relate to physicists' reference frames.
So, what is really in a rigorous mathematical language a reference frame? And in using that definition, what will be then one inertial reference frame?
 A: Here is how Sachs and Wu do it in General Relativity for Mathematicians (1977).
They start by defining a spacetime to be a connected 4-dimensional, oriented, and time-oriented Lorentzian manifold $(M,g)$ together with the Levi-Civita connection $D$ of $g$ on $M$.
Then they define an observer in $M$ to be a future-pointing timelike curve $\gamma$ such that $|\gamma_*|=1$. (This latter condition is just for convenience.)
Finally, they define a reference frame on a spacetime $M$ to be a vector field, each of whose integral curves is an observer.
The inertial frames are then those that are parallel, i.e. constant with respect to the covariant derivative. Intuitively, those are the frames that aren't "accelerating." (This is also related to Killing vector fields.) 
But on Einstein-de Sitter spacetime, for example, there ARE no parallel reference frames: that's why inertial frames don't make sense, in general, in relativity. Asking for a parallel frame is a strong constraint on the metric.
A: Let $T$ be a $4$ dimensional vector space with a non-degenerate bilinear symmetric form, $B$, with signature $(3,1)$, this is called the Minkowski space-time and its bilinear form is our space-time ruler (ruler+clock). 

A reference frame in SR is just a choice of basis of $T$ modulo multiplication by (real) scalars. 

(I'm not sure if the quotient I'm taking is the standard one or wether it can be weakened, but it certainly works).
Next consider all the elements of $GL(V)$ that leave our measurements invariant, i.e., the $g\in GL(V)$ such that $B(gv,gw)=B(v,w)$ for all $v,w\in V$. This elements form a subgroup called the orthogonal group and denoted as $O(3,1)$. Physicists know this group as the Lorentz group.
What's most interesting is how the reference frames of two different observers relate! Say, how does a spaceship traveling at constant speed in vacuum sees (measures) things in comparison to how I do it (assuming I'm in an inertial frame).
This is where the group $O(3,1)$ comes into play. Remember that the action of the orthogonal group is transitive in the unitary sphere of a vector space (maybe we need to restrict to a subgroup here). So, for every other equivalence class of a basis of $T$ we can find an orthogonal transformation which takes us to it. 
In order to explain the weird things we observe in SR you may need to take into account the projection operators w.r.t. different bases.
A: In my opinion (I'm a physicist as formation) a reference frame is only a way to give components to vectors ( and tensors or other more complex stuffs), possibly choose in a nice way such that a given problem become simpler.
Let me explain. A vector is a geometric or physical entities defined as an '' oriented segment''. This is good, but if we want perform calculations about it we need his components, i.e. numbers that we can use to find sums, products, derivatives etc...
To do so we define some reference frame in a suitable structure in whic the vector is located. This means that we can fix a reference frame in a vector space, or an affine space, or a more complex structure as the tangent space of a manifold, depending to the nature of the problem.
Mathematically this simply means thet we are using an isomorphism with a suitable structure.
Physically may be important to use particular frame, as an inertial one, because the laws of physics are simpler in such a system, or a system in a Lorentz's manifold wen we use relativity, but this simply means that we have to select a particular structure and suitable isomorphism.
