vaathi.dev

Transformers in plain language

Attention as “what to focus on,” plus encoder/decoder intuition — no heavy math.

Most modern language (and many multimodal) models are built on an architecture nicknamed the transformer. You don’t need the equations — you need the job it does.

Attention: what to focus on

When you read “The trophy didn’t fit in the suitcase because it was too big,” you decide what “it” refers to. Attention is a learned way for the model to weigh which earlier words matter for the next prediction — a soft spotlight, not a single hard rule.

AnimationAttention = which words matter for the next prediction
Attention spotlightSentence tokens appear in a row. A soft spotlight moves between trophy and suitcase while resolving the pronoun it, then settles on trophy.The trophy didn’t fit in the suitcase because it was too big.trophyfitsuitcasebecauseitSpotlight weight shifts — here “it” leans toward trophy more than suitcase.No equations — just “what should I focus on?”

That “look around the context” trick is why transformers handle longer phrases better than older one-word-at-a-time designs. The next lesson, how transformers work, walks tokens → embeddings → attention → stacked layers in more detail.

Encoder / decoder (intuition only)

Rough roles you’ll hear:

  • Encoder — Reads and represents the input (useful for understanding / classifying a passage).
  • Decoder — Generates the next tokens (chat and completion models lean heavily here).
  • Encoder–decoder — Classic for translation-style “read this, write that.”

Many chat models you use are decoder-style (often decoder-only) generators with clever training. The names matter less than the idea: represent context, then generate.

See also: Glossary — Transformer, Attention.

What you can ignore for now

Matrix multiplies, layer counts, and paper acronyms. They matter for builders. For Foundations, remember: transformers = context-aware prediction engines powered by attention. (Those multiplies are also why GPUs show up later in the path.)

Takeaway

A transformer is a design that lets the model decide what to pay attention to in the input while predicting what comes next.

Try this

Paste a short paragraph into a chat model and ask: “Which words would you need to resolve the pronoun in sentence two?” You’re practicing the attention idea — spotlighting context — without any code.