I am searching for some groups, where it is not so obvious that they are groups.
In the lecture's script there are only examples like $\mathbb{Z}$ under addition and other things like that. I don't think that these examples are helpful to understand the real properties of a group, when only looking to such trivial examples. I am searching for some more exotic examples, like the power set of a set together with the symmetric difference, or an elliptic curve with its group law.
Homological algebra. Let $A,B$ be abelian groups (or more generally objects of an abelian category) and consider the set of isomorphism classes of abelian groups $C$ together with an exact sequence $0 \to B \to C \to A \to 0$ (extensions of $A$ by $B$). It turns out that this set has a canonical group structure (isn't that surprising?!), namely the Baer sum, and that this group is isomorphic to $\mathrm{Ext}^1(A,B)$. This is also quite helpful to classify extensions for specific $A$ and $B$, since $\mathrm{Ext}$ has two long exact sequences. For details, see Weibel's book on homological algebra, Chapter 3. Similarily many obstructions in deformation theories are encoded in certain abelian groups.
Combinatorial game theory. A two-person game is called combinatorial if no chance is involved and the ending condition holds, so that in each case one of the two players wins. Each player has a set of possible moves, each one resulting in a new game. There is a notion of equivalent combinatorial games. It turns out that the equivalence classes of combinatorial games can be made into a (large) group. The zero game $0$ is the game where no moves are available. A move in the sum $G+H$ of two games $G,H$ is just a move in exactly one of $G$ or $H$. The inverse $-G$ of a game $G$ is the one where the possibles moves for the two players are swapped. The equation $G+(-G)=0$ requires a proof. An important subgroup is the class of impartial games, where the same moves are available for both players (or equivalently $G=-G$). This extra structure already suffices to solve many basic combinatorial games, such as Nim. In fact, one the first results in combinatorial game theory is that the (large) group of impartial combinatorial games is isomorphic to the ordinal numbers $\mathbf{On}$ with a certain group law $\oplus$, called the Nim-sum (different from the usual ordinal addition). This identification is given by the nimber. This makes it possible to reduce complicated games to simpler ones, in fact in theory to a trivial one-pile Nim game. Even the restriction to finite ordinal numbers gives an interesting group law on the set of natural numbers $\mathbb{N}$ (see Jyrki's answer). All this can be found in the fantastic book Winning Ways ... by Conway, Berlekamp, Guy, and in Conway's On Numbers and Games. A more formal introduction can be found in this paper by Schleicher, Stoll. There you also learn that (certain) combinatorial games actually constitute a (large) totally ordered field, containing the real numbers as well as the ordinal numbers. You couldn't have guessed this rich structure from their definition, right?
Algebraic topology. If $X$ is a based space, the set of homotopy classes of pointed maps $S^n \to X$ has a group structure; this is the $n$th homotopy group $\pi_n(X)$ of $X$. For $n=1$ the group structure is quite obvious, since we can compose paths and go paths backwards. But at first sight it is not obvious that we can do something like that in higher dimensions. Essentially this comes down to the cogroup structure of $S^n$. There is a nice geometric proof that $\pi_n(X)$ is abelian for $n>1$.
The set of exotic differentiable structures on the $n$-sphere in any given dimension is a group under the operation of connected sum, with the standard sphere being the identity element. Not at all obvious that this forms a group! For example, in dimension 7, this group is isomorphic to $\mathbf{Z}/28$.
I was surprised to learn about the elliptic curve groups. You fix constants $a$ and $b$ take the set $S$ of points on the Riemann sphere (that is, the complex plane plus a point at infinity) of the form $$y^2 = x^3 + ax + b.$$ Then define the sum two points $p_1, p_2$ on this curve by taking the straight line through $p_1$ and $p_2$ and finding the third point $p_3 = \langle x_3, y_3\rangle$ where the line intersects $S$. Then $p_3^{-1} = \langle x_3, -y_3\rangle$ is the group sum of $p_1$ and $p_2$. It's not immediately clear that there is necessarily a point $p_3$, but there is, with suitable treatment of tangents and of the point at infinity. It's not immediately clear that the operation is associative, but it is. The point at infinity is the identity element, and the inverse of the point $\langle x, y\rangle$ is $\langle x, -y\rangle $.
I was surprised the first time I saw the group of unit arithmetic functions under Dirichlet convolution. Arithmetic functions are functions $f:\mathbb{N}\rightarrow F$, where $F$ can be any field (but usually $\mathbb{C}$). The operation is $$(f\star g)(n)=\sum_{d\mid n}f(d)g\left(\frac{n}{d}\right).$$ So, here the identity is the function $$\varepsilon(n)=\left\{\begin{array}{lcl}1&:&n=1\\0&:&\text{otherwise}\end{array}\right.$$while inverses are defined recursively, as described here under "Dirichlet inverse." Note that $1/f(1)$ appears in the definition of the inverses, so we must include only arithmetic functions for which $f(1)$ is invertible in $F$ (this is why we say unit arithmetic functions).
The Brauer group of a field is not obviously a group in two ways: first it's not obvious that the group is closed under its group operation, and then it's still not obvious that inverses exist.
1- Fourier analysis: the set of non-vanishing absolutely convergent Fourier series is a group under pointwise multiplication.
The neutral element is the constant function equal to $1$. And product stability follows from Cauchy product. These are straightforward. The existence of inverses is less obvious.
Wiener's lemma: if $f(t)=\sum_{\mathbb{Z}}c_ne^{int}$ is absolutely convergent, i.e. $\sum_{\mathbb{Z}}|c_n|<\infty$, and does not vanish, then $\frac{1}{f(t)}$ is also the sum of an absolutely convergent Fourier series.
This is not that hard either. But it was, and remains, striking. You can find a proof here. Less elementary, but much more interesting, the proof given by Gelfand which raised the interest in Banach algebras. Indeed, the absolutely convergent Fourier series form a unital commutative Banach algebra with spectrum $[0,2\pi]$. More precisely, the characters are the point evaluations $f\longmapsto f(t_0)$. The invertibility of non-vanishing elements is then obvious via Gelfand representation.
2- von Neumann algebras: for a type $\rm{II}_1$ factor von Neumann algebra $M$, i.e. an infinite-dimensional noncommutative probability space, we can make sense of $t\times t$ matrices over $M$ for every real $t>0$. This gives rise to another type $\rm{II}_1$ factor $M^t$.
In their seminal work dating back to the 1930's, Murray and von Neumann introduced the fundamental group of a $\rm{II}_1$ factor $M$ $$ \mathcal{F}(M):=\{t>0\;;\;M^t\simeq M\}. $$ It is not hard, but not obvious per se, to see that this is a subgroup of $(\mathbb{R}^+,\;\cdot\;)$.
One of their striking classification results says that, up to isomorphism, there exists a unique approximately finite-dimensional type $\rm{II}_1$ factor $R$. As a consequence, it follows that $$ \mathcal{F}(R)=\mathbb{R}^+. $$ By Connes' groundbreaking work (1976), any amenable type $\rm{II}_1$ factor is isomorphic to $R$. So these also have fundamental group equal to $\mathbb{R}^+$. This includes the group von Neumann algebra $L(\Gamma)$ of any countable amenable group $\Gamma$ with infinite conjugacy classes.
On the other hand, Connes proved in 1980 that the fundamental group of $L(\Gamma)$ is countable when $\Gamma$ has Kazhdan's property (T). But it remained open for some time whether the fundamental group of a $\rm{II}_1$ factor could be trivial.
In a more recent breakthrough, Popa exhibited in 2001 such examples. In particular, he showed that $$ \mathcal{F}(L(\mathbb{Z}^2\rtimes \rm{SL}(2,\mathbb{Z})))=\{1\}. $$ On the opposite side, he also proved in 2003 that any countable subgroup of $\mathbb{R}^+$ arises as the fundamental group of some type $\rm{II}_1$ factor. For a larger class of such groups and open questions, see these slides by Vaes, another important contributor to the theory.
Finally, note that Voiculescu's free probability theory allowed Radulescu to prove that $$ \mathcal{F}(L(F_\infty))=\mathbb{R}^+ $$ for the free group on a countably infinite number of generators $F_\infty$. Unfortunately, these techniques have not permitted to compute the fundamental group of $L(F_n)$ for the free group on $2\leq n<\infty$ generators. Note that the following puzzling long-standing question remains open: $$ L(F_2)\simeq L(F_3)\;? $$
It's not obvious that the collection of integers mod $p$, excluding the coset of $0$, form a group under multiplication. In particular, it's not obvious that inverses exist. You typically use the Euclidean algorithm for that.
After some study, it might become clear that the following is a group. But it seemed nonobvious the first time I saw it: the collection of all fractional linear transformations from $\mathbb{C}\cup\{\infty\}$ to itself, with formulas $$z\mapsto\frac{az+b}{cz+d}$$ such that $ad-bc\neq 0$ (in order to guarantee that you don't have a constant map), using composition as the group operation. Firstly, it takes at least a full second to believe that the composition of two such things is another such thing. Secondly, at some point you realize that any one fractional linear transformation has infinitely many representations: $z\mapsto\frac{kaz+kb}{kcz+kd}$. So a lot of your early thoughts on the topic are not 100% correct. Thirdly, associativity is no fun to confirm directly. (Again, it's not immediately obvious, but eventually you can see that this is a factor group of the $2\times 2$ general linear matrix group.)
May be the groups of nimbers would fit the bill? The underlying set is that of non-negative integers $\mathbb{N}$. The group operation (denoted by $+$) is defined recursively as follows
$$
a+b=\operatorname{mex}\left(\{a'+b\mid a'<a\}\cup \{a+b'\mid b'<b\}\right).
$$
Here $\operatorname{mex}(S)$ is defined for all proper subsets of $\mathbb{N}$ and means the smallest non-negative integer not in the set $S$ (=Minimum EXcluded number). So $0+0=0$ simply because both sets on the r.h.s. are empty. But then $0+1=\operatorname{mex}(\{0\})=1=1+0$, $1+1=\operatorname{mex}(\{1\})=0$,
$0+2=\operatorname{mex}(\{0,1\})=2=2+0$, $1+2=3$, $2+2=0$ et cetera.
The operation is well defined, because the sets on the r.h.s. are obviously finite, and hence proper subsets of $\mathbb{N}$, for all $a,b\in\mathbb{N}$.
Now, it turns out that this operation is just the NIM-sum (addition in base two without carry). That is not entirely obvious even though it isn't exceedingly hard to see either.
It turns out that the sets of the form $S_n:=\{x\in\mathbb{N}\mid x<2^n\}$ are subgroups. Furthermore, if $n$ is a power of two, this set also has a multiplication that turns it into a field. The construction is due to Conway. See this wikipage for more.
Here are some examples:
The ideal class group of a number field $K$. It is not obvious that this is a group, because for example to be able to invert an ideal requires the definition of an invertible ideal and showing that $\mathfrak{p}\mathfrak{p}^{-1} = \mathcal{O}$. This last part is not trivial and if memory serves me right you need to invoke the Nakayama Lemma.
The fundamental group of a topological space $X$ - not trivial to show that the operation of taking products of loops is associative. When I took such a class my lecturer drew some pretty pictures to show homotopies between $f \ast (g \ast h)$ and $ (f \ast g) \ast h$, but I was not entirely convinced.
What about the set of line bundles over a manifold? This forms a group, whether your line bundles are real or complex. The difference between them is also very interesting, as one is 2-torsion, and the other can be torsion-free!
Here the group operation is tensor product: showing it is a closed associative operation is fairly easy. The trivial bundle is the identity. Ah, but what are the inverses?
Here is an example that is a bit different, namely one that appears as a subgroup of another group, but where it is not obvious that it is a subgroup.
Let $G$ be a finite group with a proper non-trivial subgroup $H$ such that for all $g\in G\setminus H$ we have $H\cap H^g = \{1\}$ (such an $H$ is called a Frobenius complement in $G$ and if $G$ has a Frobenius complement it is called a Frobenius group).
Define $$N = \left(G\setminus\bigcup_{g\in G}H^g\right)\cup\{1\}$$
Then $N$ is a subgroup of $G$, but I am not aware of a proof of this that does not involve character theory (for a proof see for example Theorem 7.2 in Isaacs' Character Theory of Finite Groups).
($N$ is called the Frobenius kernel of $G$ and it is in fact a normal complement of $H$).
Another example is mentionned here:
Let $G$ be a finite group of order $n$ and $S \subset G$ be any subset. Then $$S^n = \{s_1s_2 \cdots s_n \mid s_1, s_2, \dots, s_n \in S\}$$ is a subgroup of $G$.
Exotic example can be rubik's cube group with cube moves.
Sandpile groups have the curious feature that the identity element is very complicated. See "What is a Sandpile?" (Levine, Lionel and Propp, James. Notices of the AMS, 57 #8 (September 2010) pp. 976–979).
I always found the fact that braid groups are groups at all quite interesting. The elements of the group are all the different braids you can make with, say, $n$ strings. The group operation is concatenation. The identity is the untangled braid. But the fact that inverses exist is not obvious.
The Jacobson radical.
Take a noncommutative ring $R$ with 1. Any left ideal is either contained in another or is maximal. The elements common to all maximal left ideals, i.e. $$ J = \bigcap_i M_i, $$ is a group in two ways:
It is not obvious from their definition that KK-groups actually have inverses.
I think one of the most surprising results of this type is the following (and I could just be very naive): Kervaire and Milnor showed that diffeomorphism classes of oriented exotic spheres form the non-trivial elements of a finite abelian group under the connected sum for dimension not equal to $4$.
Two examples:
The set of algebraic numbers is a field and it isn't trivial to prove that their sum and multiplication comply to give two groups, in one hand. In the other, consider the Grothendieck group's construction.
If $G$ is a Lie group, with multiplication operator $m:G\times G\rightarrow G$ and inverse $i:G\rightarrow G$, then $TG$ is also a Lie group, with multiplication operator $Tm:TG\times TG\rightarrow TG$ and inverse $Ti:TG\rightarrow TG$. To see this, use the obvious notation that for $v_g\in T_g G$ and $h\in G$, $v_g\cdot h = T_g R_h(v_g)$, where $R_h:G\rightarrow G$ is right multiplication by $h$, and similarly for $h\cdot v_g$ (these operations are easily shown to be associative). Then for $g,h\in G$ and $\xi,\zeta\in \mathfrak{g}$, the Lie algebra of $G$, $$T_{(g,\,h)}m(\xi\cdot g,\,\zeta\cdot h) = \xi\cdot g\cdot h + g\cdot\zeta\cdot h = (\xi + \mathrm{Ad}_g\zeta)\cdot gh,$$ and so under the bijection between $\mathfrak{g}\times G$ and $TG$ given by $(\xi,\, g)\mapsto \xi\cdot g$, $TG$ is just $\mathfrak{g}\rtimes_{\mathrm{Ad}}G$, the semidirect product of $\mathfrak{g}$ and $G$ with respect to the adjoint action $\mathrm{Ad}$.
Similar to how we turn the natural numbers into the additive group of integers, and the group of integers to the multiplicative group of the rationals, let $A$ be a set with an abelian operation and an identity (abelian monoid). For $A \times A$, declare $(a_1, b_1) \tilde{} (a_2, b_2)$ iff there is a $c \in A$ such that $a_1 + b_2 +c = a_2 + b_1 + c$.
Well if you want to get more visual, you can go with symmetry groups and wallpaper groups.
Let $k$ be a field, and let $A$ be a finite type $k$-algebra.
Consider the isomorphism classes of dualizing complexes over $A$.
Given two dualizing complexes $R$ and $S$, define their "product" to be the isomorphism class of the Hochschild cohomology complex of their tensor product over $k$:
$R\cdot S := RHom_{A\otimes_k A}(A,R\otimes_{k} S)$.
Then it is not clear that:
Yet, all these turn out to be true. See Section 4 of http://arxiv.org/abs/1401.6678
This paper begins describing, in its abstract, a group operation over the set of joint distributions of $k$ random variables in a probability space with given properties. This group is denoted as $$(\mathcal{G}_k,\boxtimes).$$
A subsemigroup of a group is not necessarily a group, but a nonempty subsemigroup of a finite group is a group.
Here was an interesting example for me:
Let $X_1,X_2$ be compact $n$-manifolds with boundary. We say that they are cobordant if there exists an $n+1$ dimensional manifold with boundary so that $\partial Y=X_1 \sqcup X_2$ (this is an equivalence relation that we will denote by $\sim$.
In this case, we let
$$\mathcal{R}_n:=\{\mathrm{ compact \, manifolds}\}/\sim$$ be our base space. There is in fact a group structure $+: \mathcal{R} \times \mathcal{R}\to \mathcal{R}$ given by $([X],[Y]) \mapsto [X_1 \sqcup Y]$
Maybe unsurprisingly, the identity is $[\emptyset]$, but here is the weird part: $[X \sqcup X]=\partial (M \times [0,1])$ so in fact, $[X]+[X]=[\emptyset]$, and each element is its own inverse (idempotent.) Really, we can turn this whole thing into a $\mathbb Z_2$ vector space, and use cartesian product as a multiplicative structure to obtain a graded algebra, but I think the groups are interesting enough in their own right.
I'll list below the first few from wikipedia:, starting with $\mathcal{R}_1$ and increasing in dimension:
$$\mathbb Z/2, 0, \mathbb Z/2, 0, \mathbb Z/2 \oplus \mathbb Z/2, \mathbb Z/2.$$
The set of reversible Turing machines forms a group: https://arxiv.org/pdf/1603.08715.pdf
But, since they define Turing machines not in the classical way, but in a way that composition is an operation and then take the reversible ones, it doesn't feel like it is so difficult to see that it is a group.
I find it very interesting, thou!
At page 6-7 of Yuri I. Manin's Cubic forms is given an example of abelian group over the set of non singular points of an irreducible cubic curve in a projective plane (over an arbitrary field).
A set of permutations closed under composition.
These were historically the first thing called groups, every group algebra is instantiated by some set of permutations closed under composition and most of the theorems you see in group theory books have been developed to understand permutations closed under composition better and their relationship with solving algebraic equations.