What is an "invariant form" of a group? I have often seen this phrase used in at least two frequent contexts,


*

*One uses the notation of $\omega_{AB}$ (the matrix $\{ [0 , I],[-I,0]\}$) to denote the symplectic form for $USp$ group. 

*One uses the notation $\epsilon_{AB}$ (the matrix $\{[1,0],[0,-1]\}$) for $SU(2)$ 
I would like to know what is the precise definition and meaning of this. 
 A: A slightly strange thing about your question is that it has several answers: one for each type of classical group (symplectic, orthogonal, and unitary).  Instead of trying to understand "invariant form" by itself, it might be better to concentrate on the type of group first.  If you want to focus on the form, search for "Polar geometry".
The case of symplectic groups and orthogonal groups in characteristic not 2 is probably the simplest. For a matrix $g$, $g^T$ denotes the transpose. An invariant form for a matrix group $G$ is a matrix $A$ such that $gAg^T = A$ for all $g \in G$.
If $A$ has the property that $A^T=A$, then the set of all invertible matrices $g$ such that $gAg^T = A$ is called the orthogonal group preserving the invariant symmetric form $A$.
If $A$ has the property that $A^T=-A$, then set of all invertible matrices $g$ such that $gAg^T = A$ is called the symplectic group preserving the invariant symplectic form $A$.
If $A$ is more general, then you get an intersection of the symplectic and the orthogonal groups, and I think this intersection has a name, but I've never used it.
The case of the unitary groups requires two fields like $\mathbb{R} \leq \mathbb{C}$ or $\mathbb{F}_q \leq \mathbb{F}_{q^2}$. In each case, there is a field automorphism called (complex) conjugation such that $\overline{ \bar z} = z$ and $\overline{z+w} = \bar z + \bar w$ and $\overline{zw} = \bar z \bar w$ and if $z$ is in the small field, then $\bar z = z$. This defines a new type of transpose, $g^H$ which is the transpose of $g$ with all of its entries (complex) conjugated.
If $A$ has the property that $A=A^H$, then the set of all invertible matrices $g$ such that $gAg^H = A$ is called the unitary group preserving the invariant Hermitian form $A$.
Some people prefer forms to be functions $B$ from pairs of vectors to a scalar. Then the groups are defined as all $g$ such that $B(gv,gw) = B(v,w)$. Here $B$ is some sort of bilinear: either symmetric bilinear, alternating=symplectic bilinear, or Hermitian bilinear = sesquilinear. To convert $A$ to $B$ is easy: $B(v,w) = v^T A w$. The reverse is easy as $A_{ij} = B(e_i,e_j)$.
This makes the last definition a little easier: if $Q$ is a function from vectors to scalars such that $Q(\lambda v) = \lambda^2 Q(v)$ and $B(v,w) = Q(v+w) - Q(v) - Q(w)$ defines a bilinear form (here $\lambda$ is a scalar, and $v,w$ are vectors), then $Q$ is called a quadratic form. The matrices $g$ such that $Q(gv) = Q(v)$ form the orthogonal group preserving the invariant quadratic form $Q$. This works in all characteristics.
