Information

by Neil Rickert

People use the word “information” in a variety of ways, some of them mutually inconsistent. In this post, I shall describe how I prefer to use the term. My own usage is partly informed by my study of perception. And, of course, as a retired professor of computer science, my usage is partly informed by the use of information in technology.

The word “information” is used a lot in information technology. We talk about information being communicated or copied or transmitted. But we also think of newspapers as being sources of information. Likewise, human speech is normally considered to be information.

Shannon information

Claude Shannon is famous for his theory of information. He actually called it a theory of communication. People use the expression “Shannon information” in somewhat inconsistent ways. I shall try to stay close to what Shannon discussed.

Shannon was concerned with the question of transmitting information over a noisy channel, and with the problem of minimizing the errors that are due to noise in the channel. Shannon considered the use of both analog signals and digital signals. But the bulk of his work has to do with digital signals, and that will be my main emphasis in this discussion.

Shannon did much of his work at Bell Labs. And, of course, a major concern of Bell Labs was with telephone communication. These day, largely due to the work of Shannon and others, most telephone communication is done by encoding voice signals in a stream of binary digits. It turns out that this kind of digital communication can better deal with noise in the channel.

As I shall use the term, “Shannon information” will refer to a sequence of symbols transmitted in a communication channel. With most of information technology, those symbols are the binary digits 0 and 1. But human speech can also be considered to be Shannon information, where the words used are the symbols. And, for printed text, we can consider the printed letters to be the symbols.

I don’t want to say much about symbols. However, we should note that they are categories. If we think of printed text, then a variety of shapes of ink mark can all be considered to be the same letter. We talk of using different fonts to allow those different shapes. But we treat all examples of the letter “a” the same, regardless of font, unless our concern is with artistic values. So symbols, as I am using the term, are just categories that our culture has accepted to use in communication.

When we use speech or printed text, we are normally interested in expressing something meaningful. However, meaning plays very little role in Shannon’s theory. If I have a meaningful sentence, then I can communicate that sequence of words. The meanings play no role in transmission of the symbols (the words). And as long as I can get the sequence of words to the other end of the transmission channel, the message receiver can usually determine the intended meaning. The meanings themselves are not transmitted. Only the symbols are transmitted.

Linguists often distinguish between syntax and semantics. Here “syntax” refers to any rules for combining words, while “semantics” refers to the intended meanings. Because Shannon’s theory has only to do with the words, or other symbols, Shannon information is sometimes described as syntactic information.

I do not plan to say much more about Shannon’s theory. If you are looking for that, check the Wikipedia link that I provided near the beginning of this post.

Semantic information

When we humans communicate, we are particularly interested in the meanings of what is being said. When I talk of “semantic information”, my concern will mainly be with communicating meaning.

In one sense, Shannon information is semantic information. Shannon’s theory is a theory of communication, and there would be no point in communicating unless there was meaningful content in the message. But, looked at differently, there’s no such thing as semantic information. What is transmitted in the communication channel is just symbols or syntactic units. Meaning itself is not part of what is transmitted.

The meaning comes from us. The message sender produces a sequence of symbols (such as words). And they are transmitted. The channel itself knows nothing about the meanings of these words. But the message receiver does know how to recognize the intended meaning.

This mostly depends on shared knowledge. The sender and receiver share knowledge of the intended meanings of words used. The sender uses that knowledge to generate a suitable sequence of words. The receiver uses that knowledge to recognize the intended meaning of that sequence of words. It is broadly agreed that this shared knowledge amounts to a system of social conventions (meaning conventions) within a language community.

Meanings and categories

I have already pointed out that the symbols used are categories. They are categories of abstract entities (the symbols). But if we look at meaningful communication, as with semantic information, we notice that our meaningful statements are all statements about categories. In the commonly use illustrative sentence “the cat is on the mat”, both “cat” and “mat” are categories of real world things. And “on” is a relational category. Our meanings are closely related to our categories.

We acquire meanings when we learn to categorize our world. And I have been suggesting that such categorization is central to perception. Knowing a language requires knowing the shared conventions of meaning. And, in turn, this requires knowing the shared ways that our society categorizes.

If we listen to songbirds, their sounds seem to meet the requirements to be Shannon information. But we are not able to easily decipher what messages they are sending. That’s because we do not have the required knowledge of the categories that these songbirds use. And without such categories, we cannot make sense of their apparent communication.

Perception and semantic information

There’s another way that people use the expression “semantic information”. According to some view, when we look around we can just see that the world is full of information. And all we have to do is pick up that information. That seemed to be an assumption of J.J. Gibson, who described visual perception as picking up information from the optic array.

In my view, this is nonsense. There is no semantic information lying around for us to just pick up. Since semantic information is expressed in terms of human categories, the human independent world would have to somehow be generating information consistent with human categories. To me, this does not make sense.

It is true, however, that when we look around there seems to be a lot of information available. But we do not look at the world in itself. What we look at, is our perceived world. Our perceptual systems are in the business of categorizing the world. The semantic information that we think we observe is really being generated for us by our perceptual systems.

Our perceptual system is, in effect, digitizing (or discretizing) the world for us. Neural pulses are, in effect, reporting categories to us. Hebbian learning, where the neurons adjust how they trigger, is really a way of fine-tuning or calibrating how our perceptual system categorizes the world. Perception is a semantic engine, and not a syntactic engine. AI (artificial intelligence) system, by contrast, are mainly syntactic engines and they are not well suited to dealing with semantic questions.

Summary

That’s roughly how I am looking at “information”.

%d bloggers like this: