Self-information

People like you are child sponsors

Within the context of information theory, self-information is defined as the amount of information that knowledge about (the outcome of) a certain event, adds to someone's overall knowledge. The amount of self-information is expressed in the unit of information: a bit.

By definition, the amount of self-information contained in a probabilistic event dependens only on the probability that the event happens. More specifically: the smaller this probability is, the larger is the self-information associated with receiving information that the event indeed occurred.

Further, by definition, the measure of self-information has the following property. If an event C is composed of two mutually independent events A and B, then the amount of information at the proclamation that C has happened, equals the sum of the amounts of information at proclamations of event A and event B respectively.

Taking into account these properties, the self-information H(A) associated with event A that has a probability is defined as:

bits. This definition, using the logarithm function, complies with the above conditions.

This definition can be rewritten as:

(bit).

Examples

On tossing a coin, the chance of 'tail' is 0.5. When it is proclamed that indeed 'tail' occurred, this amounts to

bits of information.

When throwing a die, the probability of 'four' is 1/6. When there is proclamed that 'four' has been thrown, the amount of self-information is bits.
When, independently, two dice are thrown, the amount of information associated with {throw 1 = 'two' & throw 2 = 'four'} equals 'throw 1 is two & throw 2 is four'throw 1 = 'two' & throw 2 = 'four' bits.
This outcome equals the sum of the individual amounts of self-information associated with {throw 1 = 'two'} and {throw 2 = 'four'}; namely 2.585 + 2.585 = 5.170 bit.