Reinventing Entropy | Compression is Intelligence Part 1

Summary by Krater

An exploration of Shannon's information theory and entropy, examining how compression is mathematically equivalent to prediction and how the Shannon entropy theorem establishes limits on text compression.

From the video

Answers: What is the connection between data compression and information theory according to Claude Shannon?

• information theory
• Shannon entropy
• data compression
• prefix codes
• cross-entropy
• language models

What it concludes

• Prediction and compression are mathematically equivalent, representing two sides of the same coin.
• No encoding scheme can compress data beyond the limit set by Shannon entropy.
• The information content of an event is equal to the negative logarithm base two of its probability.