# I don't understand the concept of “Information content of a sentence” [closed]

I came across this concept in my course literature. Here's a snippet from that chapter:

"An essential component in logical reasoning is how much information a sentence contains. If S is a sentence, the number of 1's in the truth table for S can be seen as a measure of the information content in the sentence. The more 1's a sentence has the less information the sentence contains"

## closed as unclear what you're asking by Rob Arthan, Holo, Yanko, amWhy, max_zornSep 15 at 0:37

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

• Of course, if there are no 1's at all in the truth table, then per the answer below, that's not any information at all: it's a contradiction, which lacks sense. So the relation breaks down at zero 1's. – elliot svensson Sep 14 at 17:29
• What is the context here? Truth tables are usually used in propositional logic. Propositional logic is completely neutral as to the significance of truth or falsehood: by negating a proposition, I can swap the 1s and 0s in its truth table without making any difference to its information content under an assigment of truth values to the propositional variables. The claim that information content is essential to logical reasoning also seems quite bizarre. – Rob Arthan Sep 14 at 20:44
• @TiStrga: your link is an interesting introduction to Shannon's information theory, but doesn't help much with this question, which is about some unexplained notion of the information content of a propositional formula measured by the numbers of 1s in its truth table. – Rob Arthan Sep 14 at 20:52
• @RobArthan The information content of an event is certainly not neutral with respect to negation; it's the negative log of the probability of the event, so it goes down as the probability (in this case, the number of 1s) goes up. You may be thinking of the information entropy, which here would be better applied to a question "is this statement true or false" rather than the statement itself: it's the expected information content of an answer to the question. – Misha Lavrov Sep 14 at 22:34
• @MishaLavrov: I think we may be talking at cross-purposes. The question is phrased in terms of the "information content of a formula" and not the "information content of an event" and hence doesn't make sense to me. The OP needs to supply a reference and/or give more details about what he or she does not understand. – Rob Arthan Sep 14 at 22:40

I assume you are asking for an intuitive explanation.

Imagine the police are looking for a killer, and there are only 20 remaining suspects, one of which is missing an arm. There are several witnesses. One says:

This sentence is not very informative, right? Why not? It's because it is true for 19 of 20 suspects (it has $19$ ones in the truth table).

On the other hand, if the witness says

The killer was a man

and half the suspects are male, then that is a more informative sentene (it has $10$ ones in its truth table), and if a witness says

The killer was 180cm tall, red haired, wore glasses and a ring on his right hand

then it's quite likely that there is only one one in the truth table of that sentence, which is therefore very informative.

• Great answer! I'm a little curious if you meant to use the fact that one of the suspects is missing an arm. Can't see why else you included that detail unless it was going to come back later, but then it didn't – Jared K Sep 14 at 15:32
• @JaredK: Suppose that instead, the witness said “The killer had 1 arm.” Despite its parallel construction, this sentence contains more information than the sentence “The killer had 2 arms“, because of its lower probability. – Dan Sep 14 at 16:01

4.46 Among the possible groups of truth-conditions [i.e. lines in the truth table] there are two extreme cases.

In one of these cases the proposition is true for all the truth-possibilities of the elementary propositions. We say that the truth-conditions are tautological.

In the second case the proposition is false for all the truth-possibilities: the truth-conditions are contradictory.

In the first case we call the proposition a tautology; in the second, a contradiction.

4.461 Propositions show what they say: tautologies and contradictions show that they say nothing.

A tautology has no truth-conditions, since it is unconditionally true: and a contradiction is true on no condition.

Tautologies and contradictions lack sense [i.e. they convey no "information"].

[...] (For example, I know nothing about the weather when I know that it is either raining or not raining.)

This is an odd claim. If you have a statement $S$ that depends on $n$ inputs $x_1,x_2, ... x_n$, then a truth table will have $n+1$ columns (one for each $x$, and one for $S$), and it will have $2^n$ rows, one for each combination of $x$ values. Each $x$ will be True half the time and False half the time. So there will be $(n+1)2^n$ total entries, of which $n2^{n-1}$ will definitely be True, and $n2^{n-1}$ will definitely be False. The remaining $2^n$ entries will depend on $S$. So first of all, only $\frac1{n+1}$ of the entries will even depend on $S$ to begin with.

Of those $2^n$ entries, the amount of information is symmetric with respect to True and False; having $a$ True and $b$ False is just as informative as having $b$ True and $a$ False. If $S_1$ is always True, and $S_2$ is always False, then knowing that $S_2$ happens to be False in a particular circumstance is just as uninformative as knowing that $S_1$ is True.

"Information" is generally defined in terms of entropy. The entropy of a particular possibility is the negative log of the probability of that possibility. The total entropy of a random variable is the sum of the individual entropies, weighted by the probabilities. So

entropy = $-\sum p_i\log(p_i)$

If there are only two possibilities, then if we represent the probability of one as $p$, then the probability of the other is $1-p$, thus the entropy is $-p\log(p)-(1-p)\log(1-p)$. This is maximized when $p=\frac 12$, and minimized when $p=0$ or $p=1$.

So a statement is most informative on average when the probability of it being True is 50%. If the probability of it being True is very low, then the information content when it is True is very high, but since that won't happen very often, its average information content is low. For instance, a statement with probability 50% has entropy 1 bit. A statement that has probability $\frac 1{16}$ of being True has 4 bits of entropy when True, and .093 bits when False. Since False happens more, that entropy is weighted more, so the total entropy (weighted average) is .223 bits, less than one fourth the entropy of a statement that is True half the time.

So if we interpret "number of 1's in the truth table" as "number of times True appears in the S column of the truth table" and "information the sentence contains" as "entropy when the statement is True", then this claim makes sense. But it could be much more clear.

• Entropy is only defined for a random variable, and information content for an outcome of a random variable. So information content precisely is "information when a statement is true", and entropy is the expected information content. – Misha Lavrov Sep 16 at 15:29
• To use the example in the accepted answer: the question "Does the killer have 2 arms?" has low entropy, because even though the information content of the answer "no" is very high, that answer is very unlikely, so the average information content of an answer to the question is low. – Misha Lavrov Sep 16 at 15:30
• @MishaLavrov In lay use, "statement" implies that the content of the statement is claimed to be true, but in logic, a statement is something that can be true or false. In essence, a "statement" is simply a boolean variable that is possibly made up of/is a function of other boolean variables. Thus, your distinction between the "question" and the statement is on shaky ground. I stand by my conclusion, given in the last paragraph of my answer: under a certain interpretation, it's true, but it could be a lot clearer. – Acccumulation Sep 17 at 14:42