How to measure the information content of a function?

Question

I would say that the function $f(x)=x^2$ has more "information" than the function $f(x)=1$. Is this true, and if so, is there a way to measure the information content of a function?

Answer 1

I'd suggest that the Kolmogorov complexity (https://en.wikipedia.org/wiki/Kolmogorov_complexity) of your $f(x)$ is the best way to define/measure its information content. It's simply the length of the shortest computer program that calculates $f(x)$. And then, clearly, $f(x)=1$ can be represented by a shorter (though not by much) program than $f(x)=x^2$.

This is actually a pretty common interpretation of information content of a function. A more elaborate discussion is at https://www.mdpi.com/1099-4300/13/3/595/htm (and google kolmogorov complexity vs entropy, or similar query, for more).

Answer 2

I would say that information is a relative thing. You need to define a reference space, so you can talk probabilities. When you have probabilities, then you can measure information.

If your space contains only the two mentioned functions, then you need to make a single measurement at $x=0$ to decide which function is presented. So, the functions are equal and you will need a single bit to describe the space.

You may have a different space though. For instance, take the same functions, plus a few new functions that are exactly similar to $f(x)=x^2$, on an interval $[a,b]$ and mutually different, on the rest of the domain. Then, we need more information to pinpoint $f(x)=x^2$, compared to $f(x)=1$, as there are some similar functions.

Answer 3

To be concrete, let $f:[-N,N]\rightarrow \mathbb{R}.$ The usual way of defining the distribution of a random variable is how we can proceed. Assume the uniform measure on $[-N,N]$ with $N<\infty$, and let $f(X)=Z.$ Then $$\mathbb{P}(Z \in A)=\mathbb{P}(f(X) \in A)=\mathbb{P}(f^{-1}(A)).$$

In this context we can say that the entropy of the function is the entropy of its output as a random variable.

If we choose the $f(X)=c,$ some constant then clearly $H(Z)=H(f)=0,$ since all the probability is concentrated at one point.

If $f$ is one-to-one, then the entropy is maximal.

If $f$ is two-to-one, then the entropy is smaller $(f(x)=x^2).$

Since it is the entropy of the function which maps the set $[-N,N]$ I wouldn't put any other measure on $[-N,N]$ and that's why I took $N<\infty$ to ensure the uniform distribution exists.

Answer 4

a function has no information. it can only destroy information.

so what you are trying to measure is perhaps how much does function ''preserve'' information, rather than how much a function ''creates'' information.

one way to quantify the information preservation of a function is the following guessing game: given a function's output, what can you say about its input that has resulted in this output?

take f(x) = 1. this function destroys all information. For f(anything) = 1, by observing 1 as the output, you gain 0 insights on what input it could have been.

take f(x) = x^2. this function destroys some information. For f(-1) = f(1) = 1, by observing 1 as the output, you can say the input has to be either -1 or 1.

take f(x) = x. this function preserves all information. For f(1) = 1 is the only possible explanation of why the output is 1.

With this in mind, one can define certain entropy term that reasonably measures the information preserving properties of the function:

draw an output y uniformly at random from the set of possible outputs, and let X be the random variable takes on the value x such that f(x) = y. By measuring the entropy of X, you can quantify how well a function ''preserves'' information: the lower the entropy of X, the better function f is at preserving information.

and we arrive at a similar conclusion to @kodlu's answer, but hopefully in a more intuitive sense