Algebraical "thing" having two inverses per element? I know in groups we often demand one unique inverse and it brings some particular properties. But what would we call something algebraic where each element has two different inverses?
Does this happen in any algebra? If so, what does it lead to?

Imagine in New York starting at some crossing we go s7a4 : seven streets up and 4 avenues up. 
Then inverse (in the sense of taking us back to our original position) could be either s-7a-4 or a-4s-7 depending on if we want to go downwards streets or avenues first. It is completely reasonable to consider these different things as we encounter different stuff on the way. Different amount of traffic, different shops, different bars, and so on.
 A: That can happen in a semigroup: a set with an associative product and (in your case) an identity.
For example, consider the set of finite sequences of unit left and right steps on the line, where multiplication is concatenation. The identity is the empty sequence. Then $p$ will be an inverse of $q$ just when it has the same number of left (right) moves as $q$ has right (left).
You can do this on a grid to model your New York City example. (It's a more interesting challenge if you allow yourself to walk diagonally along Broadway.)
Edit in response to (correct) comments.
It is indeed  the action of these operations that matters. The quotient structure that captures the result of an action in this example is just the additive group of the integers acting on the integers: all that matters is the difference between the number of right and left steps. 
So more formally I think the algebraic structure the OP is looking for is that of a semigroup acting on a set, not just a semigroup.
A: Two inverses cannot happen in groups. If $b,c$ are inverses of $a$, i.e., $ab=1=ba$ and $ac=1= ca$, then
$$b = 1\cdot b = (ca)b = c(ab) = c\cdot 1 = c.$$
A: For groups you do not need to demand uniqueness. That just follows from the existence.
The only thing that comes to my mind is something like the following:
Let $X$ be a non-empty set and let $Y$ be an arbitrary set. One can show that a map $f \colon X \rightarrow Y$ Is injective iff it has a left-inverse, i.e. there exists a map $g \colon Y \rightarrow X$ such that $g \circ f = \text{id}_X$. To construct such $g$ one takes preimages for all elements in $f(X)$ and sends the other elements of $Y$ to some arbitrary elements in $X$. Therefore these at least do not have to be unique. That is not a real inverse though.
One can of course also do the same for right-inverse maps.
I think the objects that you are searching are called loops or are related to them. I would just browse the wikipedia page for magmas to maybe find what you are searching for.
