I am not a mathematician by any means, so please be clement if my question happens to be naïve.
In many textbooks, matrices make their first appearance in the context of a discussion of linear transformations of vectors. Then the author usually proceeds to consider changes of basis, deduce formulae like $T^{-1}AT$, and so on. So far, so good, except for the fact the term «matrix» actually disguises the tensor nature of linear transformation.
Matrix is just a representation of tensor in this context. But what is important here is that despite their somewhat abstract nature, tensors correspond to physical values. However, the notion of covariance has no meaning when one talks about matrices. That makes it possible to consider matrices as in some sense as more general objects than tensors (speaking simplistically «every tensor is a matrix, but not every matrix is a tensor»; yes, tensors can be of any rank, but let's talk about 2D case only).
Many mathematical objects are intimately connected to the properties of physical objects. Of course, by definition, they are generalizations and abstractions, but this intimate touch with reality persists nonetheless. Here are a few examples of such connections: continuity — topology; counting — ordinary numbers and fractions; rotation — complex numbers and quaternions; the rate of change — derivative; linearity — tensors (vectors) and so on. I would not go too far and say that every mathematical object should have some physical meaning, even in principle. It is maybe that matrices are an example of such «pure mathematical machinery.» But, if so, how can it be that they appear in many physical equations (well, they pervade all mathematical physics!)? It looks like, for example, Pauli of Dirac matrices should have some meaning. And this is my question: can matrices (not the objects they represent) be associated with any property of the world.