Representation of a group The elements of a transformation group are usually represented by linear operators acting on a linear vector space (LVS). We usually represent the elements of a group as square matrices acting on (the elements of) a LVS. Can't we use other linear operators to represent the elements of a group?
 A: Not all linear representations can be given in terms of matrices (unless you consider infinite dimensional matrices). For instance unitary representations of Lorentz group cannot be given in terms of matrices, since all these representations work on infinite dimensional Hilbert spaces (it is related with the fact that the group is not compact). So the used operators are unitary operators.  As I said, if you fix a Hibertian basis, in principle,  you may represent these operators as infinite dimensional matrices, but I cannot really imagine a situation where such a cumbersome  representation could have some technical interest, for the Lorentz group at least.
Dropping the requirement of unitary operators, you may also have finite dimensional representations of the Lorentz group where operators can be represented as matrices (obviously, once you have fixed a basis). The simplest case are spinorial representations.
Restricting to deal with matrix representations (and this is the case for irreducible unitary representations of compact groups), the used matrices have necessarily to be square matrices, because they have to be invertible as the elements of the represented group are invertible.
