Finding $d$ and $r$: I don't know whether this is more or less clumsy than your existing algorithm: to write $n$ as $dQ+r$ for a given $Q,$ you can compute $x=nQ^{−1}.$ Then compute the floor of the elements of $x$ to get $d,$ and compute $n−dQ$ to get $r.$ To keep the arithmetic integer, if desired, you can write $Q^{−1}=\mathrm{adj}(Q)/\det Q.$ Since the adjugate matrix, $\mathrm{adj}(Q),$ is integer, the vector $y=n\,\mathrm{adj}(Q)$ is integer, and $d$ can be obtained by dividing $y$ by $\det Q$ using integer division. This algorithm applies in dimensions higher than $2$ as well.
To verify that this works, the first key idea is that $x\mapsto xQ$ is a linear map. So $(ax+by)Q=a(vQ)+b(uQ)$ for any vectors $x$ and $y$ and scalars $a$ and $b.$ This means that the unit square spanned by $e_1=(1,0)$ and $e_2=(0,1),$ that is, the set $\{ae_1+be_2\mid 0\le a,b<1\},$ is sent to your parallelogram, that is, to $\{au+bv\mid 0\le a,b<1\},$ under this map.
The second key idea is that $Q$ is invertible. Consider a point $n=dQ+r,$ where $d$ is an integer vector and $r$ is in your parallelogram. Multiplying by $Q^{-1}$ from the right on both sides, we get $nQ^{-1}=d+rQ^{-1}.$ To show that the algorithm works, that is, to show that the floor of this expression equals $d,$ we need to show that all components of $rQ^{-1}$ lie in the interval $[0,1)$. But we have established that $Q$ sends the unit square to your parallelogram. Since $Q^{-1}$ is invertible, it reverses the process, sending points in the parallelogram to the unit square, which is what we needed to show.
The intuition is that $dQ,$ with $d$ an integer vector, is a lattice point. If your parallelogram is translated to this lattice point, the other three vertices of the translated parallelogram are $(d+e_1)Q,$ $(d+e_2)Q,$ and $(d+e_1+e_2)Q.$ Points in the interior of the translated parallelogram are $xQ,$ where, writing $d=(d_1,d_2),$ $x=(x_1,x_2),$ we need $d_1<x_1<d_1+1,$ $d_2<x_2<d_2+1.$ So the floor of $x_1$ and $x_2$ are $d_1$ and $d_2.$
Where do $P$ and $Q$ come from? I'm not sure from your question whether you are starting from the set $P$ and obtaining $Q$ from it, or starting from $Q$ and computing $P$ from it, or whether both $P$ and $Q$ are given data. It seems to me that if you need to compute $P$ from $Q,$ that's a slightly tricky problem—harder, in my opinion than computing $d$ and $r$ from $Q.$ (One can always use the algorithm for finding $r,$ combined with brute force, but for a very long and thin parallelogram this might not be efficient.)
Note that the same lattice can be described by infinitely many different matrices $Q.$ Specifically, a matrix $Q'=QU,$ where $U$ is an integer matrix with determinant $\pm1,$ produces the same lattice as $Q$ does. Matrices like $U$ are called unimodular matrices. The product of two unimodular matrices is a unimodular matrix, so unimodular matrices form a group. This is essentially the modular group (up to questions about whether determinant $-1$ is allowed, and whether $U$ and $-U$ should be considered equivalent).
In your position, however, I suspect a study of the modular group might not be the most direct approach to answering the questions you're interested in. I would focus more on the computational, algorithmic literature. I don't have a good recommendation, unfortunately, but you might start by reading up on the LLL algorithm. Even if you don't have an immediate use for that algorithm, it is likely that someday you will. But more to the point, you'll probably pick up a lot of useful insights about how people who compute with lattices think about things.
To finish the discussion above: if you use $Q'$ instead of $Q,$ you have a different parallelogram, and therefore a different set of points—call it $P'.$ You can create a dictionary between $P$ and $P',$ using the algorithm for finding $r.$ (Note that $P$ and $P'$ will have the same size; correspondingly, $Q$ and $Q'$ will have the same determinant, up to sign.)
Fundamental region not a parallelogram: If you want to do the same thing for a hexagonal tiling, say, you can proceed in a similar way. (There's a bit of a terminology issue here: a hexagonal tiling is not a lattice: mathematically, a lattice is a group, which requires that the sum of two lattice vectors be a lattice vector. This is not true of the vertices of a hexagonal tiling.) Note that in a parallelogram tiling, the four vertices are equivalent. So $P$ contains one vertex and a bunch of interior points. In the hexagonal case, the six vertices are not all equivalent; there are two equivalence classes, consisting of three vertices each. So $P$ will contain two vertices and a bunch of interior points.
You can always define a parallelogram-shaped fundamental region that gives the same thing as the hexagonal one. Then use the algorithm above for finding $r.$ Finally you will need a dictionary mapping the points of the fundamental parallelogram to the points of the fundamental hexagon. Devising an algorithm to produce such a dictionary may be a bit awkward, but once you have the dictionary, things are straightforward.