# Information Theory of Deep Learning. Bottleneck theory

The Information Bottleneck Method by Tishby, Pereira, and Bialek is an interesting way to look at what is happening in a deep neural network. You can see the concept explained in following papers

- Deep Learning and the Information Bottleneck Principle , and
- ON THE INFORMATION BOTTLENECK THEORY OF DEEP LEARNING

Professor Tishby also does a nice lecture on the topic in

Information Theory of Deep Learning. Naftali Tishby

With a follow-on alk with more detail:

Information Theory of Deep Learning - Naftali Tishby

Deep learning aside, there are other interesting applications of the bottleneck method. It can be used to categorize music chords:

Information bottleneck web applet tutorial for categorizing music chords

and in this talk, the method is used to quantify prediction in the brain

Stephanie Palmer: "Information bottleneck approaches to quantifying prediction in the brain"

I found the following talk also interesting simplified version of the concept applied to deterministic mappings:

The Deterministic Information Bottleneck