Representation theory is a broad field that studies the symmetries of mathematical objects. A representation of an object is a way to "linearize" that object as a group of matrices. It's the non-commutative analog of classical Fourier transforms.

Representation theory is the study of mathematical objects via their symmetries. The broad idea is to study all the ways to "linearize" an object by embedding it in a ring of matrices. This reduces problems in abstract algebra to problems in linear algebra, which are better understood.

Specifically, given some algebraic object $A$, a representation of $A$ is a vector space $V$ and a a structure-preserving map $\rho\colon A \to \mathrm{End}(V)$, the ring of endomorphisms of $V$. That is, we're embedding $A$ inside a ring of matrices to "linearize" $A$. The usual first example that students see of representation theory is in the case of finite groups, where $A$ is a finite group and $\rho$ is a group homomorphism.

Studying representations of a ring can also be thought of as studying modules over that ring. For a ring $R$ and a vector space $M$, a representation $\rho \colon R \to \mathrm{End}(M)\colon r\mapsto \rho_r$ gives $M$ the structure of a left $R$-module, with action $r\cdot m = \rho_r(m)$.

Representation theory generalizes Fourier analysis to harmonic analysis, and is also used in the study of automorphic forms in number theory.