Given a matrix $X$ of dimensions $n*d$ my goal is to determine the amount of resources used by this matrix. The definition for resources used here is flexible, but relates to the amount of information encoded in the matrix. That is, a a completely uniform matrix uses very little resources as it can be compressed quite easily.
This relation between resources used and information contained reminded me of entropy. However, my field of specialization is computing hence I'm wondering if there's some better concepts I'm missing. The problem with calculating entropy is that it requires the distribution of the random variable $X$ (here my matrix entries) however I do not have access to such a distribution, nor can I reasonably build one given the single matrix (in the context of my problem).
I also thought of SVD, and how information contained can be related to the variance of the data, however I moved away from that solution as measuring the amount of resources used would require me to know the true distribution (ie. the mean and variance when 100% of information is conserved) which I do not have.
What would be the best way to quantify the amount of resources used, or the amount of information used by/contained in a matrix?
Edit: Some context around my problem. I am developing a machine learning algorithm which seeks to build the optimal architecture for a neural network given some resource constraint. In my setting the resource constraint is amount of information encodable, equivalently, space available for the model to use in memory. To do this however I need to calculate the amount of resource the model uses. It isn't sufficient for me to say it's taking a certain amount of memory because that tells me nothing about the actual content of the information. Information in a neural network is encoded in a weight matrix, where there is one weight for each layer. $X$ is one such weight matrix.
Edit 2: The neural network weights are can be seen as an approximation to the real distribution of the inputs. We don't have the actual distribution, so this is our best guess. Information here is meant to be the relative difference between the unknown true distribution and our best guess distribution. I believe one could relate this to the free energy concept in a restricted Boltzmann machine. My goal here is to say, I have encoded $Z$% of the true distribution in my approximation, and this approximation can be stored in $Y$ amount of bits at best. So I know how much resource this information takes.