Views: 1
BERT and GPT are often described as fundamentally different architectures. The actual difference in the attention computation itself is a single additive term. Here’s what that term does, and why it changes everything about how the model can be used.
BERT and GPT get discussed as though they’re built from fundamentally different ideas — “encoder-only” versus “decoder-only,” bidirectional versus autoregressive, understanding versus generation. The framing is accurate at the level of capability, and slightly misleading at the level of mechanism. Strip both down to their attention computation, and the difference is a single additive term in one equation.
The formula both architectures share
Scaled dot-product attention computes, for every position in a sequence, a weighted blend of every other position’s information — the weighting determined by how relevant each position is to the one currently being processed. That’s the entire computation, and it’s identical in both an encoder and a decoder block: same query/key/value projections, same softmax, same scaling.
The one line that changes everything
The difference is a mask, added to the raw attention scores before the softmax step: for every pair of positions where the second comes after the first in the sequence, add negative infinity. Since softmax exponentiates its inputs before normalizing, and the exponential of negative infinity is exactly zero, this doesn’t discourage attending to future positions — it makes it structurally impossible. Every future-position weight comes out to exactly zero, every time, regardless of what the raw similarity score happened to be.
Add that one term, and a block that could freely attend in both directions — well suited to building a complete understanding of an entire, already-known sequence — becomes a block where each position can only see itself and what came before it, which is exactly the constraint a model needs if it’s going to generate text one token at a time, since at generation time the future tokens simply don’t exist yet.
Why this single term explains so much downstream behavior
This is the actual mechanism behind capabilities that otherwise seem like separate design choices. An encoder-style model, unmasked, can look at an entire sentence at once when deciding how to represent any single word — which is exactly why encoder-based models excel at tasks like classification and understanding, where the complete input is available up front. A decoder-style model, masked, is architecturally prevented from doing the same thing — which is exactly why it’s suited to generation, and exactly why it processes prompts left-to-right rather than holistically. Neither behavior needed a separate mechanism invented for it. Both fall directly out of the presence or absence of one additive mask in an otherwise identical computation.
Why this matters practically
Understanding this collapses a lot of apparent architectural complexity into one simple fact: the encoder/decoder distinction that shows up throughout transformer literature isn’t two different families of mechanism requiring separate mental models. It’s one mechanism, with a single binary choice — mask the future or don’t — determining which of two very different classes of task the resulting model ends up well suited for. When evaluating or debugging a transformer-based system, that’s often a more useful question to ask than “which architecture is this” — it’s “is attention here allowed to see the future or not,” because that one property, more than almost anything else about the model, determines what it can and can’t be used for.
Leave a Reply