← All Labs
INTERACTIVE APPLET · INFORMATION THEORY · ENTROPY

Entropy as a Guessing Game

I've hidden one symbol. You may only ask yes/no questions — how many will it take to pin it down? If every symbol is equally likely, you are stuck splitting the possibilities in half again and again: eight equally likely symbols cost three questions, sixteen cost four. But if the deck is lopsided — one symbol shows up almost always, the rest are rare — a clever questioner asks about the common one first and usually wins in a single question. The average number of yes/no questions a perfect strategy needs is exactly Shannon's entropy, measured in bits. Skew the distribution and watch two things move together: the entropy \(H\), and the tree of questions the optimal Huffman code builds to reach it. Rare events carry more bits because they are more surprising — and surprise, averaged, is information.

H = − Σ p log₂p  =  average yes/no questions to name a symbol

The optimal question tree — split by probability, not by count

Skew the distribution · Huffman rebuilds the tree · entropy is the average depth
Canvas · live Huffman coding

DRIVE IT

Question node (a yes/no split) Symbol (a leaf, its code = path) Current guess path Surprisal −log₂p
Left edges read "yes", right edges "no". A symbol's codeword is the string of answers from the root down to it — and its length is how many questions it takes to reach. Frequent symbols sit shallow; rare symbols sit deep.
ENTROPY · H (bits)
the ideal average question count
0.00

DISTRIBUTION

Skew 0 = uniform (all equally likely). Higher = one symbol dominates and the rest grow rare.

PLAY

PRESETS

WALKTHROUGH

  1. Start with the preset Fair 8. All eight symbols are equally likely, the tree is perfectly balanced, and every codeword is 3 bits long. Entropy reads exactly 3.00 — you cannot do better than three yes/no questions.
  2. Now drag Skew up. The distribution tilts: one symbol swells, the others thin out. Watch the Huffman tree rebuild live — the common symbol floats up to a shallow leaf (a 1-question win), the rare ones sink deep.
  3. Read the entropy clock as you drag. It drops below 3.00. A predictable source is easier to guess, so it needs fewer questions on average — even though the deepest rare symbol now costs more than 3.
  4. Switch to ② PLAY THE GAME and hit Auto-play. Each trial draws a hidden symbol and walks the question tree; the running average questions (teal) converges onto the dashed entropy line. The game literally measures \(H\).
  5. Open ③ SURPRISE. Each symbol's bar is its surprisal \(-\log_2 p\): rare symbols tower, common ones barely register. Entropy is just the average height, weighted by how often each symbol actually appears.

AHA

THE INSIGHT
Information is surprise made countable. A symbol that arrives with probability \(p\) carries \(-\log_2 p\) bits — one bit for every halving of the odds — so a coin-flip event costs 1 bit and a near-certain one costs almost nothing. Entropy is the average surprise, and by Shannon's theorem it is the unbreakable floor on the average number of yes/no questions, or code bits, needed to name the outcome. The Huffman tree you see is that floor, drawn: split by probability, not by count.

RESEARCH NOTE

Shannon's source coding theorem (1948) brackets the optimal average codeword length \(L\) by \(H(X)\le L< H(X)+1\); Huffman coding (1952) achieves the minimum \(L\) for symbol-by-symbol codes, and coding long blocks drives \(L\to H\). The "20 questions" reading is exact when all \(p_i\) are powers of \(1/2\), where \(L=H\) precisely. Further reading: Shannon, A Mathematical Theory of Communication; Cover & Thomas, Elements of Information Theory, ch. 2–5; the Kraft–McMillan inequality \(\sum 2^{-\ell_i}\le 1\).

EXPLANATION

In 1948 Claude Shannon asked a deceptively childish question: how much information is in a message? His answer reframed information not as meaning but as reduction of uncertainty, and this applet turns that abstraction into a game you can play. A source spits out one symbol at a time — letters, pixels, DNA bases, coin flips — each with some probability \(p_i\). You do not know which symbol is coming, and your only tool is the yes/no question. The number of questions a perfect strategy needs, averaged over many draws, is the entropy of the source, and it is measured in bits — one bit being the information in a single fair coin flip.

The building block is surprisal. If an event has probability \(p\), its surprisal is \(-\log_2 p\) bits: an event that always happens (\(p=1\)) is no surprise and carries \(0\) bits, a fair coin (\(p=\tfrac12\)) carries exactly \(1\) bit, and a one-in-a-million event carries about \(20\) bits. The logarithm is the whole trick — it makes information additive, so the surprise of two independent events is the sum of their surprises, exactly as two independent coin flips carry two bits. Entropy is then the expected surprisal, \(H(X)=-\sum_i p_i\log_2 p_i\): you weight each symbol's surprise by how often it actually turns up. Toggle mode ③ and you are looking straight at this sum — tall bars for rare symbols, short bars for common ones, and \(H\) is their probability-weighted average height.

Why does the guessing game land on the same number? Each well-chosen yes/no question ideally splits the remaining probability into two halves, so one question removes one bit of uncertainty. A symbol of probability \(p\) needs about \(\log_2(1/p)\) such splits to isolate — precisely its surprisal. The Huffman algorithm in the DRIVE IT panel builds the optimal question tree by a greedy rule: repeatedly merge the two least likely items into a combined node, and repeat. The effect is that frequent symbols end up on short branches near the root (few questions) and rare symbols on long branches (many questions). A symbol's codeword is just the sequence of yes/no answers on its path, and Shannon's source coding theorem proves the average length of any such code cannot beat \(H\): \(H\le L< H+1\). Huffman hits the lower edge of that band, and coding symbols in long blocks squeezes \(L\) all the way down to \(H\).

This is why entropy is the master quantity behind data compression, and why it appears everywhere once you look. A file compresses well exactly when its symbol distribution is lopsided — low entropy, predictable, few questions per symbol — and a stream of true random noise cannot be compressed at all, because at maximum entropy every symbol is equally surprising and every question buys a full bit. The same \(H\) sets the capacity of a communication channel, quantifies the diversity of an ecosystem, measures the disorder of a thermodynamic system (Boltzmann's \(S=k\ln W\) is the same idea in different units), and defines the loss a machine-learning model minimizes when it does "cross-entropy" training. The lesson of the game is compact: you cannot ask fewer questions than the world is uncertain — and entropy is exactly how uncertain the world is, in bits.