Here’s something that blew my mind recently: compression and language modeling are, at their core, trying to solve the exact same problem. Once you understand this connection, large language models stop feeling like magic and start feeling inevitable — like a mathematical truth that was always there, waiting to be scaled up.
The Compression-Prediction Equivalence
At its most fundamental level, compression is about finding patterns. If you can predict what comes next in a sequence, you can compress it — because you don’t need to store the parts you can predict. The better your prediction, the smaller your compressed output.
This isn’t a new insight. Claude Shannon figured this out in 1948 when he founded information theory. The optimal compression of a sequence is directly tied to how well you can model the probability distribution that generated it. But the practical implications for AI are profound.
How Compression Works
Consider the simplest form of compression: run-length encoding. If you have AAAAABBBCC, you can write it as 5A3B2C. You’ve predicted that the data contains runs of repeated characters, and exploited that pattern.
More sophisticated compressors like LZ77 (the basis of gzip) look for repeated substrings. Instead of storing a sequence twice, they store a reference: “copy 10 bytes from position 50.” The compressor is predicting that patterns in the data will repeat.
Huffman coding goes further: it assigns shorter codes to more frequent symbols. It predicts which symbols will appear more often and optimizes accordingly.
LLMs Are Compression Engines
Now here’s the leap: a large language model is doing exactly the same thing, just at a much more sophisticated level.
When an LLM predicts the next token in a sequence, it’s building a probability distribution over what comes next. That prediction IS compression — if you can predict the next token with 90% accuracy, you only need ~0.47 bits to encode it instead of the full ~11 bits for a token from a 50,000-word vocabulary.
The training process — minimizing cross-entropy loss on next-token prediction — is literally minimizing the number of bits needed to encode the training data. The model is learning to compress the entire internet.
This is why scaling works. More parameters mean better compression, which means better prediction, which means better generation. The relationship isn’t coincidental — it’s mathematical.
The Practical Implications
Understanding this equivalence has real engineering consequences:
1. Quantization Is Lossy Compression
When you quantize a model from FP16 to INT8 or INT4, you’re applying lossy compression to the model’s weights. The same principles apply: you’re trading precision for size, and the question is how much information you can discard before the model’s predictive ability degrades significantly.
2. Prompt Caching Is Dictionary Compression
When an LLM caches a prompt’s KV state, it’s doing something analogous to dictionary compression — storing a computed representation so it doesn’t have to be recomputed. The cached state IS the compressed form of the prompt’s context.
3. Context Length Is Window Size
The context window of an LLM is like the sliding window in LZ77. A larger window means more opportunities to find patterns and make predictions, but it costs more memory. The entire field of efficient attention mechanisms (sliding window, sparse attention, etc.) is about expanding the effective window without proportional memory costs.
4. Temperature Is Coding Noise
When you set temperature > 0 in an LLM, you’re adding noise to the probability distribution — essentially choosing a suboptimal code. This is why high-temperature output is more creative but less coherent: you’re deliberately using a worse compressor.
The Deep Connection
The compression-prediction equivalence also explains something that puzzles many people: why can LLMs do things they weren’t explicitly trained to do?
Because compression requires understanding structure. To optimally compress English text, you need to understand grammar, semantics, world knowledge, reasoning patterns, and everything else that makes language predictable. There’s no shortcut — the optimal compressor of human language must understand human language.
This is also why the Turing Test and compression tests are related. A system that can perfectly compress any text you give it must, by definition, understand that text at a deep level.
What This Means for the Future
If LLMs are compressors, then the path to better AI is clear: better compression. This means:
- Better training data — more diverse, higher-quality data means more patterns to learn
- Better architectures — more efficient compression of those patterns
- Better inference — faster decompression (generation) at runtime
The next time someone asks you how LLMs work, you can give them a satisfying answer: they’re the world’s most sophisticated compression algorithms, trained on the internet, and they work because compression and understanding are the same thing.
This article is based on Compression is prediction by Annie Sexton, published on the ngrok blog.
답글 남기기